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ABSTRACT 

At the 1977 Edacational Testing Service (ETS) 
Invitational Conference, ^the ETS Heasurewent Award was presented to 
Anne Anastasi. In view of the convergence of leasur^went and thd law, 
the conference focufi^d ou six related issues. Barbara lerner explored . 
the s<;reening^ procedures of Aierican professional and graduate 
schools in "Equal Protection and External Screening: Davis, De "Funis, 
and Bakke,** and was responded to by Ernest M. Bernal, Jr., and Deane 
C. Sieaer. Belvin B. Novick discussed funding allccaticns uader Title 
I, adiissions policy and the Bakke case, federal guidelineiB for 
eapld^ent testing, and due 'process in the handling of suspected, 
cheating cases, in "The .Influence of the* Law cif Professional 
Measure went Standards. Win ton H. Banning responded to Novick* s 
presentation ^/in theifinal session, Vayne H« Holtzwan reviewed the 
iwplicationil of several, court cases in ^Validity and Legality;" 
Charles L. Thowas discussed' **Sowe Possible Social Iwplications of 
Recent^ Court Decision^;** Norwan Frederiksen offered soae ideas for 
iaprovewent in test use in "There Ought to Be a Law;" a^d Bichael 
Scriven suggested what seasureMent experts light learn abbiit decision 
■aking frow the law, in "The Logic of Judgwent in Evaluation and the 
Law Baking Hard Decisions with Soft Data." (EW) 

* Reproductions supplied by EDRS are the best that ^an be wade 4^ ♦ 

* ^ • , * frow the original docuwent^ ♦ 
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Foreword 



Educational Measurement and the Law, the theme of the 1977 ets 
Invitational Conference, .was timely for several reasons. Public 
interest in the Bakke case is an insistent reminder of a new fact: that 
what people do in the field of measurement or what uses they make 
of the results of measur^jment are no longer the concern only oPthe 
testing fraternity or indeed of educators or employers. These actions 
have entered the realm of public concern and debate. 

In view of this convergence of measurement and the law, the 1977 
Invitational Conference was intended to bring together peoplq with 
expertise in both these areas to discuss some of the critical issues. 
The conference focused on six aspects of measurement and the law 
in three session?. Barbara Lerner, the first speaker of the morning, 
explored the screening procedures of American professional and 
graduate schools and the conflicts that have arisen as a result of our 
^effons to maintain the rights of both individuals and of groups. In 
the second session, Melvin N^ovick took a close look at five critical 
areas of educational measurement and the ways in which they are af- 
fected by the law and suggtisted a number of actions that might be 
taken to better deal with those situations. 

Two of the afternoon speakers were concerned with technical and 
social aspects of the theme. Wayne Holtzman, in his paper on 
Validity and Legality,'* reviewed a number of court cases and dis- 
cussed their concern with several typts of test validity. Charles 
Thomas focused on some of the reasons for legislation designed to 
increjNe opportunities for minofities and the concerns expressed by- 
proponents and opponents of such affirmative .action programs, in 
his paper, Tlwfe Ou^lwio Be a La\^\ Norman Frederiksen offered 
some clew-neaded and practical ideas for change and improvement 
in t^ie use «f ([ests and testing. Michael Scriven. thcfinjil speaker of 
* the day, suggested vvhat the field of measurement might learn from 
the law abouVthe complex processes of decision making. 

It wa^s a rich day with stimulating presentations for which wt^are 
most graf'cful not on^ly loj^c speakers Ynentrgned above but also to 
Ernest BernaK Deanne SiemeP, thaddc'us Holt, and Winton Mann- 
ing, whose discussions of the morning presentations added art im- 
portant crfjical dimension io the conferent(^and to Roger Lcnnon 
and Alfred Fif^ whb p/esi(,ied over the second and third morning 
sessions. • \ . . 



• Looking back at the topics discussed at the 1977 Invitational 
Conference, one is reminded again of the truth of E. B. Whitens pre- 
diction of a bright future for complexity. It is perhaps a mark of our 
maturity as a field that we are beginning to confront that complexity 
more squarely. 

WilUam W. lurnbuU 

PRESIDENT 

Educational Testing Service 
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The ETS Award for Distinguished Service ta Measurement was es- 
tablished in 1970 to be presented annually to an individual whose 
work and career have had a major impact on developments in educa- 
tional and psychological measurement. The 1977 Award was 
presented at the 'ets Invitational Conference by President William 
W. Tucnbull to Dr. Anne Anastasi withthe following citation: 

Recognized inicrnaiionally as a leading cxponcnl of differential psy- 
chology. Anne Anaslasi has excilcd a major influence on its develop- 
ment as a behavioral science. Educational mcasuremcni has bene- 
fited significantly from hei inquiries ihicj^ihe nature of individual dif- 
ferences. 

She has confronted the controversy regarJijig the effects of heredity 
and environment on abilities, clarifying the joint role of these torrtli- 
lions in human development, and s>\eepii}g a\\ay many of the presup- 
positions, .myths, and fallacies generated by misconceptions of their 
iniexiiciions. * 

Her careful studies of the idenlifualion of psychological trails have 
encouraged the construction of instruments heller suited for mea.suring 
these lrail|i. to avoid the misleading generalities that havy frequently 
^Tch aracieriked such measures. 

Her respect for racial and cultuial diversiiy undeilies her concern foi 
appropriate consideiation of elhniL ditTercnLCs b> lest makers and test 
users. 

Her excellence as a mastei teacher is IcHeLtcd in the numerous edi- 
tions of her classic te\is. niffcn /utal /^^< //c>A;.i,m and Psu lu)lo}:ii(tl 
7 V v////^' H e r ed i t o r ill I sk i 1 1 s a re u e 1 1 d ol i mi c i\{ eii i n /V a Proble m v in 
Porspcaivi\ a collection of papers from the i rs Invitational Con- 
ference covering tvveni^ live veai< of piogress in measurement. Her 
other books and a v\ealth of articles provide theorists and practi- 
tioners uiih technical ami picfLtiJal infoimation on topics ranging from 
lest construction and statistical meiliods to studies in creativity. 

She has given freely t>.' her talent for executive leadership in hei* 
'presidencies of the American Ps\Lhological Association, the American 
Psychological l-iMuUlalion. the r.asiern Ps>Lhological Association, and 
other professional groups, and in hci Lhaiimanships of many commit- 
tee s a nd c o n fe re nee s . 

R.xprcssing the admiration and gratitude of the educational mea- 
surement conMiuinil> foi hei invaluable Lontri but ions to its body of ' 
knowledge and understanding, i is has the honor to present the 1977 
Award for Distinguished Sei vice to Measurement to Anne Anaslasi. 
« 
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Previous Recipients of ttie 
.ETS Measurement Award 

7970 £. F. Lindquist ' 

1971. LeeJ.Cronbach 

1972 Robert L. Thomdike 

1973 Oscar LBuros 

1974 J.P.Guilford 
'1975 Harold Gulliksen 

1976 Ralph Winfred Tyler 
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Equal Protection and 
External S.creening: 
Davis, De Funis, and Bakke 



BA^A ^LERNER 



ychologist and Attorney ' 



I 

External screening . is 4he process by why*h an institution^decides 
which outside applicants to accepf^and which t^ reject. Such^screen- 
ing is a necessity whenever applicants outnumber'^'Openings. In 
America today, applicants outnumber openings for most jobs arjfl. 
Jor most graduate and professional school places. Jhe inevitable, • 
result is that many are disappointed. The disappointed come in all 
colors and both sexes and they come from a wide variety of e^hnic^ 
religious, and socioeconomic backgrounds. No background creates 
indifference to rejection; none immunizes against the pain of^disap- 
pointment and the sense of inferiority and/or i*^jystice it can 
generate. * 

Scre;^ening may be inevitable; no particular method is. Institutions 
choose the screening methods they employ and their choices have . 
an ejiormaus impact on American society^ Wise choices can 
decrease feelings of inferiority and injustice for all Americans, 
regardless of ancestry: un^wise ones can increase them td dangerous 
levels. The thesis of this presentation is that American graduate and 
professional schools have made a series of unwise choices in recent 
years, as illustrated by the record in the De Funis * and Bakke^ cases ^ 
and that the founcjation for a better approach can be discerned in the 
Dav/5case.' 

All three of the above-mentioned cases deal with the relationship 
between external screening and the Equal Protection Clause of the 
Fourteenth Amendment. All have reached the Supreme Coilrt of the 
United States in this decade and all have excited an unusual amount 
of controversy and commentary. Instead of focusing directly on 
these cases, my plan is to try to illustrjate the thesis set forth above 
by presenting the following: first, a brief hjstory of the screening 
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Equal Prottctlon and External Screening 

procedures' American universities have used over the past four 
decades; ^cond, a challenge to the two principles that have come to 
dominate their approach to external screening in recent years; and 
third, a rough outline of what seems to me to be a better alternative, 

!• External Screening for Admission to Graduate and 
. Professional Schools: 
The Status Quo, Past and Present 

The first external screening case to reach tjie Supreme Court, De 
Funis V, Odegaard, involved a charge by the plaintiff, a rejected 
white male applicant, that the screening procedures used to deny 
him .admission to the University of Washington Law School^ 
violated his right to equ^l protection by discriminating against him' 
because of his race. Law schools are hardly the only institutions 
whose screening methods are being challenged these days. The 
challenged institution in the Bakke case— just argued in the 
Supreme Court— is a medical school. In the Davis case, it was not 
an educational institution at all, but a police djipartment, deciding 
which" job training applicants to accept and "which to reject. Of 
course, external screening methods vary within and betwee.; giC^'ns 
of institutions, but overall, the approaches of American universities 
have been similar enough to permit one grOup of institutions, law 
schools, to serve as a useful illustration of what most of the others 
have done in the past and are doing jiow. 

Initially, American law schools needed no external screening 
methods and u^ id none. As recently as in the 1940s, the number of 
applicants did not exceed the number of openings. Almost anyone 
with a bachelor's degree who*wanted to enter law school could and 
did. The trick then was to stay in, and it was a hard trick. Flunk-out 
and drop-out rates of over 50 percent were common/ This 
method— -nhrow them all in and let them sink or swim"— was 
harsh, particulariy on applicants who entered the pool wearing lead 
shoes but did not discover that painful fact until after a large and 
fruitless expenditure of money,, time, and effort. In,auJition, it was 
an inefficient and expensive method for the institutions involved in 
terms of wasted resources, lowered morale and performance levels, 
and high trjiining and turnover costs. The only advantage to this no- 
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screening system was that everyone who wanted a chance to try got 
it. ' 

In the fifties, the situation began to change. Increasingly, the 
number of applicants came to exceed the number of openings, and 
some forn^f external screeniog'became a necessity. Law schools- 
responded in the most reasonable wj&y possible. They adopted a test 
admirably designed to accomplish its purpose: to predict in advance 
which applicants had a reasonable chance' of successfully complet- 
ing the course of studies necessary to prepare them to function as 
competent attorneys and which did not. The l^st was the lsat*^ 
and* as a result of its widespread use, flunk-out Bnd drop-out rates 
.declined dramatically.^ . 

What the law schools were doing at this point, in effect, was 
screening ion the basis of necessary ability. The ability in qifestion, 
verbal-reasoning'skilK is obviously not the only ability necessary for 
satisfactory performance as a law student, altorney',*or judge, but it 
is, undeniably, a necessary one — a "'key element," lo use more 
technical jargon — and it is the only relevant ability that is now rela- 
tively easy to measure in an objective fashion .^This type of screen- 
ing involves the use of an absolute cut-off swre to divide,tK©se who 
have the necessary abilitysfrom those who lacleit. I call it absolute 
because it^ is a single, fixed point, a constant dividing line chosen to 
represent a necessary ability levet for a given task. It is, 6f course, 
not an absolute in the sense of b<^ing either preordained or infallible. 

Selecting, a cut-off point is, in essence, a matter of choosing the 
odds one wants to play by. Thus, if research with the lsat shows 
that applicants who score above point A have only a 50-50 chance 
of successfully completing the program at a particular law school 
whereas applicants who score above point B, have a 70 percent 
chance and applicants above point C a 90 percent chance, one is free 
to set the cut-off at an,y of those three points. As a result, administra- 
tors who adopt this approach are not donning 'strait-jackets or en- 
dorsing a minimal standard. Xonsiderable latitude for discretion 
remains. • . ^ 

The main limit is that perfection is not attainable. Any cut-off 
score will exclude some who might have made it in spite'of the odds 
against them and will include others who will not make it in spite of 
the^odds favoring them. Itx general, law schools chose to set their 
cut-off points at fairly high levels in oiUer to 'maximize the number of 
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successes and minimize the number of failures. The results, overall, 
were positive ones, for the schools, the applicants, and society as a 
whole. 

Moving from an Absolute to a Relative Standard 

Society, however, Icept on growing, in terms of size and aspiration 
rievels. As both the population and the percentage of that population 
wishmg to become lawyers increased,'' law schools faced a new 
problem: what to do when there were 2,000 applicants for 500 seats 
arfd 1:000 or them scored ab^Qve even the high cut-off pdints es- 
tablished. What the law schools did was to move from an absolute to 
a relative standard. 

What that means is that instead of deciding how high a score wqs 
necessary for competent performance in law school and equalizing 
th? chances of everyone who scored above that point, law schools 
Xhose to differentiate not only between qualified and unqualified ap- 
plicants but also between the qualified ones, selecting only those 
with thQ highest scores and rejecting those with lower, but still quite 
adequate, scores. Thus, instead of striving to meet a fixed standard, 
qualified applicants to law schools are now forced to try to score 
above a constantly shifting point in order to beat out other qualified 
candidates. How high is high enough depends. upon how high one's 
competitors manage to score in the particuhy'year iijAvhich one hap- 
pens to apply. 

tafer, an attempi^ill be made to analyze the consequences of 
this approach and to compare them with the consequences of other 
possible approaches. For present purposes, it will suffice to 
illustrate the inflationary impact of a relative standard over time. 
The LSAT lias a maximum range '0/ 800 points. In 1961, the median 
LSAT score of studentii at 81 percent of the nation's law schools was 
below 485.« In 1975, not one of 128 ABA-approved law schools had 
an entering class with a mean bdow 510. Seventy percent of them 
had means between 572 and 693.*' What this means in comparative 
terms is that most American lawyers and judges practiqing today 
v^ould. never have gotten into law school at all if they had had to 
^ compete against the inflated standards which now govern admission. 
This single, inflationary, relative-standard approach was used by 
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virtually all American law schools throughout the sixties. One result 
was "that large numbers of qualified candidates of all races were re- 
jected and made to feel intellectually inadequate, even though, in 
factothey were not. Another was that among the qualified candi- 
dates who were accepted as of 1969, only 3,8 percent were black. 
This low percentage embarrassed American law schools and, in the 
seventies, they chose to increase it by shifting to a dual relative- 
standards system.** Tl>ey did this by segregating black ^nd white ap- 
plicants and accepting only the highest scorers,within each separate 
racial ^roup. Because'the .-median score within the white group 
higher than that within this, black group, thi^ meant that many re- 
jected Aite applicants had scores above 'those pf many accepted 
blacjc applicants., - ' 

-Marco De FiHiis, a ^ould-be law student, and Allen* Bakke, a 
would-be medical student, were both in that position and they, like 
thousands of other applicants in the sa^e position, felt unjustly 
treated. Their sense of injustice is Widely shared: A recent Gallup 
Poll showed that 83 percent of the population felt that minorities 
should not be given special preferences.*- More striking still is*the 
fact that 64 percent of nonwhites agreed.*^ Nonetheless, most 
American universities have adopted a dual.-sUmdards approach* 
which was superimposed upon^'their earlier ^h^ica of a relative 
rather than ,an absolute merit standard. The result is a systeni 
characterized by two principles: duality and relativism. 

Before concluding .this description of the status quo and going on 
to challenge it, there is a ritual gestv,/e that 1 'diyi supposed to make.* 
In addition to noting that the lsa r artd other admissions tfcsts are not 
used alone but in conjunction^wjth another objective numVical incji- 
cator of ability— the ugpa, or undergraduate grade-poinl average— 
I am supposed to say something like this: ''Of course, universities 
should not rely exclusively on objective numerical indues. There 
are numerous unquantified subjective fatK)rs which also affect an 
applicant's performance in school and thereafter, and these, too, 
rightfully enter into admissions decisions." As you have doubtless 
gathered, I am not comfort^le about making this gesture. 

It is not that I wish to derogate the importance of subjective fac- 
tors. Far from it. John Dean is a more recent law, school graduate 
than either of the W<itergate special'prosecutors, Mr. Cox and Mr. 
Jaworski, so it is probable that his lsat scores were higher, but few 
among us would regard him as the better lawyer. Integrity, courage, 
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compassion, wisdom, and judgment seem to me, as I believe they do 
to most Americans, to be at least as important as the- purely in- 
tellectual abilities. 

My problems with the ritual gesture ar^e twofold. First,''it seems to 
me to be somewhat disingenuousl^Few admissions officers really 
look at subjective factors affecting those who score beloNv some 
inflktfecl quantitative qualification line, whether there be one line for 
'everybody or one for whites and another for blacks. Second, I am 
troubled by the Widespread tendency to confuse subjective factors 
with subjective methods of assessing them. In psychometric theory,, 
there is no necessary relationship between the two: Subjective fac- 
tors can and should be objectiyely identified, defined, and 
assessed-^even quantified.-Nothlng we now know suggests that that 
is an inherently impossible task. It is only a very difficult one, 
requiring sustained and systematic effort over a long period of time, 
and the sad fact is that as /et, we have not really expended that soit 
of effort in this area. Instead, what we have done is to let various^ 
collections of officials decide the fate of large numbers 6f qualified 
yaung. people on the basis of personal feelings, hunches, and biases 
in ways that are, often startingly unsystematic, inconsistent, and ar- 
bitrary, as the record irj the Bakke case seems to illustrate. 

Arbitrary standards run the risk of violating the due process 
clause as well as the Equal Protection Clause.'^ In addition, they are 
a confession -of ignorance, showing th^t we do not yet know how to 
identify and nteasure relevant subjective factors. Worse still, they 
' are a guarantee of continued ignorance. Unless we are explicit about 
. our hunches and systematic in applying' them and checking the 
results, we will nevfer learn which ones are valid and which totally 
invalid. That, and not numbers per se, is what psychometric testing, 
'and science itself, is all about. It is a method that permits us to learn 
from our mistakes sq that we do not make the same ones over and 
over again. 

/ 

li. Challenges to the Status Quo 

The Case for Abandoning Dual Standards 

Basically, the argument for dual standards is that they are necessary 
because very few minority students would be admitted without them 
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and that would be a bad thing for three reasons: 1) because it would 
deprive all students of the educational benefits of diversity; 2) would 
lessen our ability to provide adequate professional service to, all 
groups; and 3) would be unfair to the disadvantaged. In the para- 
graphs which follow, I am going to try to convince you thajl all three 
of these fine-sounding arguments are spuria^iS. 
. Let me start with diversity. As a firstgeneratiofi American who 
grew up in a multi-ethnic urban neighborhood and felt enriched by 
the diversity around me, I have a'high opiniojl of its value. I also 
have some ideas about how to achieve it withOut using any quotas or 
giving special preferences to any group, ideas I shall set forth in the 
last section of this paper. For now, the relevant point is that very 
few proponents of dual standards actu^ly seem to rknow or care 
much about diversity, about the truly remarkable variety of heri- 
tages which make the American experiment unique. 

Typically , proponents of dual standards refuse to recognize the 
fact that, we are a nation ;of Poles, Italians, Irish, Gi;eeks, Arme- 
nians, Lithuanians,. Germans, Dutch, and English, to name only a 
few.*6 Instead, they insist upon lumping all of thesediverse peoples 
together into one category which, happily,. stiir seems artificial to 
most Americans who are not members either of the KuXlux Klan or 
of^he American Nazi Party: white people. " - 

Diversity is, in fact, used as a euphemism or code word for a hand- - 
ful of currently favored minorities. I use that cumbersome phrase- 
because I know of no other way to characterize the groups^singled 
out by EEOC and included in mosprffirmative action programs.*^ In 
theory, there are four;Jn practiie, thr^e, or mere accurately, twO 
and a half. The tHgoretical four-ace these: blacks, American Indians, 
Spanish Americans, and Orientals. The practical two and a half are 
blacks, Indians, and soAie Spanish Americans but not others— 
mainly those from -Nfc^tico and Puerto Rico but not those from 
^pain, Cuba, or other Latin American countries. The latter Spanish 
Americans are excluded for the same reason that Orientals were ex- 
cluded from th(^special admissions program challenged in the De 
Funis case:** Their lsats and ugpas weVe high*WTmjgh to permit 
them to obfiin adequate representation even in the face'' of the 
grossly inflated relative standards which govern law school ad- 
missions today. 

Whether one looks at the theoretical four or the practical two and 
j8i half, itjis inaccurate to characterize them as nonwhite peoples. Ac- 
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cording to U.S. CensQs statistics. 98 percent of all Spanish Ameri- 
cans are wii!^e.*' Are tSiey then distinguished by the fact that as 
groftps» they generally score lower than members of other groups on 
tests? Not really, even if one ignores the problem posed by the high- 
scoring Japanese and other Orientals* such as thf^ Vietnamese. 
Earlier sfudies, most of them badly in need of updating.^^ suggest 
that Japanese, like Jews, lend to have higher group averages on 
many tests than White Anglo-Saxon Protestants but Jo the best of 
my knowleoige, no affirmative action official has yet suggested that , 
WASPS, >^ith l^te i^ossibie exception of Appalachians, be included in 
any special admissions program. Similarly, such statistics as we 
have suggest that as group^. Italians, for example, piay tend to score 
below WASPS, but they are not included either. A history of discrimi- 
nation does not work any better: There is no evidence proving that 
prejudice against Puerto Ricans is any greater than that agaiVi'st 
Greeks, Turks. or^^Xrabs. for example. 

One could go on and on with this listing of nonpnnciples. if time 
permitted. It doeis not. so I will simply stop and go on to the next 
argument for duaKtandards. I will also continue to use my own eu- 
phemism for currently favored minorities— Wru'Aj — partly in the 
intereist of brevity a^d partly because, like law schools, blacks 
provide a useful illustration of a more general phenomenon. 
^ Argument 2 has to do with the need tt^'provide more adequate 
professional services. This, too, is a code phrase requiring transla- 
tion. What it really means is "we need more bhK:k lawyers" and. on 
the surface, it seems a powerful argument. Thus, given the following 
statistics and asked: "Should there be more black lawyers?" most 
men and women of gobd will would probably answer "yes." The 
figures are these: Blacks constituted about 11 percent of the total 
population and 12.7 percent of those in the 21-to-25-year-old age 
range usual for law school entrants, but only 5.3 percent of all law 
students, in 1976.-* Prior to the widespread institution of dual stan- 
dards in the 1970s, they constituted an even smaller proportion of all 
law students: 1.3 percent in 1964" and 3.8 percent in 1969.'^ 

What these figures actually prove is that it is misleading to think 
about any problem in a vacuum, divorced from its context. In 
context, the question is not "Should there be more black lawyers?" 
but ''Compared to what?*' Please nole thatcthe question here has to 
do with what, not *u7/<>. The u7/o has already been decided: Only 5.3 
percent of all college gradn;Ues are black.'** and it is this 5 percent 
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that the law schools apd medical schools are fighting over. If they 
shield lose the fight, poverty and unemployment will not win it; 
otheii graduate schools and/or professions, such as physics, 
chemistry, history, education, engineering, journalism, social work< 
and business administration,*^ will. 

Whichever discipline wins, none of the 5 percent are likely to end^ " 
up as impoverished, unskilled laborers. Neither are they likelv to set- 
rich by going to law school, cleo, the Council on Legal Eduifational 
Opportunity, a group devoted to recruiting black colleg^raduates 
for the law, reports that their average black law'^school graduate 
earned between $10,000 and S 12,000 at his first posl-law school 
job." CLEO officials shrink from admitting the fact that white law 
school graduates earn considerably more^ thtm that, but^ go on 
af^way to explain the discrepancy. They suggest tharblack at- 
tornj^s prefer to work in criminal and civil rights law serving the 
poorrather than in more lucrative areas like corporate finance and 
taxation.^' Perhaps, although that is pure speculaj/on without a 
shred of supporting evidence: > 

Robert L. Tucker, the black attorney for the Reverend Jesse 
Jackson's (J()eration push, offers a differeni explanation. He con- 
cludes that black lawyers in general are not making big money be- 
rause there still aren't enough black-owned big businesses to sup- 
port black law firms, In the regrettable absence of conclusive evi- 
dence for either position, Mr. Tucker's speculations seem at least as 
worthy of serious consideration as do those of cleo officials. In 
either case, it is far from clear that black college graduates— or the 
black community as a whole— will derive greater benefits from 
careers in law than from careers in business, the sciences, and the 
arts, and that— when one cuts through all the rhetoric and the De 
Tocqueville quotes— is what the fighting on this issue is really 
about. So much for argutnent 2. 

Argument 3, that dual standards are necessary if we are to be fair 
to the disadvantaged, is even weaker. Disadvantaged students are 
those'from homes characterized by poverty and/or ignorance, and it 
is as insulting as it ivinaccurate to equate black with poor and ig- 
norant. The average black American today is nather. In fact, al- 
though a greater percentage of blacks than of whites fall into the im- 
poverished and undereducated category, two thirds of the un- 
fortunates in that melancholy position are white.-*' Such evidence as 
we have suggests that what is true for the population in general is 
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also true for law school applicants. 



In 1976, those applicants were asked this question: "Would you 
describe yourself as coming from a low income family background, 
such as from a family with a yearly income under $6,500 during youf ^ 
pre-college years?**^^ Three-quarters of the **yes** answers came 
from whites,'^* If American law schools were to give pref^^eailjal 
treatment to all students who answered "yes/' regardless 'irf 
ancestry, no court would be likely to find their action unconstitu- 
tional, That, however, is not what the law school3 have done. 
Instead, they have given preferential treatment to blacks from 
wealthy and well-educated families while denying it to whites from 
poor and poorly educated families. 

Disadvantaged, it seems, is only another euphemism for currently 
favored minorities. The sole jus^tification advanced for this sort of 
help-the-rich-at-the-expense-or-',ithe-poor policy is that blacks suf- 
fered from segregation, whites^ilid not. This was certainly true, and 
utterly deplorable, prior to ISp4 when Brown v. Board of Educa- 
tion^^ was decided. However, this happens to be 1977, and from 
this, it would seem to follow that the average 21-year-old black ap- 
plicant was bqrn in 1956 and entered first grade in 1962, eight years 
after Brouvi was decided. 

By that time, state-enforced segregation was largely a thing of the 
past. What remained then and continues today is a high degree of 
voluntary separation between all groups with distinctive heritages. 
Then, as now* many Italians — to choose one example almost at 
random — preferred to live in dominantly Italian neighborhoods, not 
because the state made them do it and not because they hated other 
Americans with different ethnic, religious* or racial 'backgrounds, 
but because they valued their distinctive cultural traditions and did 
not want to be melted down in the great Zangwillian pot or otherwise 
^ divested of tfteir identity. Neither were they rejecting the 
American dream. Rather, they, like countless other groups of 
Americans-* felt that they could makevi positive contribution as^ 
Italian-Anxericans vvithoui sacrificing either the hyphen or the word 
that preceded it. 

State-enforced segregation was an>" unmitigated evil because it 
wrongfully deprived black Americans of the choice.between assimi- 
lating and maintaining a separate group identity. Government- 
enforced integration runs the risk of again depriving black people of 
their right to choose. My own view is that the government has no 
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more right to force integration on blacks than it had to force segrega- 
tion on them. Recent decisions suggest that the current Supreme 
Court, while .remaining absolutely unyielding in its opposition to 
state^enforced segregation and discrimination, does not view the 
Equal Protection Clause as a mandate to homogenize the country. 

/The NAACP may.-be distressed because it has alwa/s been strongly • 
in favor of integration and assimilation, but many olacks do not 
agree." The,point here is not that the naacp is wrong and more 
ethnically oriented blacks are right. Rathfer, the point is that this is a 
choice that each black man and woman should have a right lo make 
for him- or herself, free of extraneous legal pressures. The blind 
insistence on condemning as inferior the education of every black 
child who did not go to a white-dominated school is^nc such ex- 
traneous pressure.^ * ' 

In fact, such evidence as we have suggests that most blacks are 
mortf* concerned about quality education than they are about integra- 
tion and that they fefuse to equate the two.^* Since we have no real 
-evidence to indicate that they are wrong" and no right to make the 
choice for them, perhaps it is time we stopped telling them ih^ their 
education was inferior whenever most of their classnj^it^ were 
black. If we do stop, then there is no basis for prefji^etfflal treatment 
for blacks qua blacks. There is a basis for pr^fefential treatment for 
poor blacks as there is for poor whites: fairness to the disadvantaged 
in the literal, not the euphemistic, sense. 

The arguments for dual standards may be weak, as I have tried to 
ilemonstrate; the arguments against them are not. First and fore- 
most, dual standards compromise the constitutional principle that 
discrimination on the basis of race violates the Equ^l Protection 
Clause. Counter arguments about benign purposes or effects miss 
the point. As the Sujjfeme Court explained in Brown, ''In the first 
cases in this CoQrtaconstruing the Fourteenth Amendment, decided 
shortly after its adoption [in 1868], the Court jnterpreted it as 
proscribing all state-imposed discrimination against the Negro 
race."^* The basic principle— that a person's constitutional rights 
cannot depend on his color— was not compromised in any \vay until 
1&% when^he Court first permitted state-imposed discrimination on 
"the basis of race, rationalizing its retreat with the ^'separate but 
equal" doctrine. 

The damage done by th?t single qualification of principle was, and 
still is, incalculable. 1; took more than half a century to recover the 
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lejgal high ground that had been lost. Slowly, painfully, beginning 
with th\f 1938 case of Missouri ex rel Gaines v. Cmada,^^ the Court 
relraced its steps until finally, in the Bmnv/'case, it removed the last 
taint of qualification* holding that "[I]n the field of public education, f 
the doctrine of 'separate but equal' has no place."*** To then turn 
around, only a little more than two decades later, and allow discrimi- 
Inatton on the basis of race, for^any purpose, is to compromise that 
prin *nle once again, to add needless complications, qualifications, 
"balancing tests," ambiguities, and loopholes which will haunt us 
ever after.. ^ 

Even the strdngest proponents of dual standards concede that at 
best, they can only be a temporary expedient.**- The weakening of a 
key constitutional principle, however, would be a permanent loss. 
Much more could and should be said about this cardinal legal issue. 
Fortunately* that has been done already, and so masterfully that the 
briefest and best advice-l can give is to urge everyone lo read the 
amicus briefs by Philip Kurlaad anJi his colleagues in the De Funis 
and Br/AAr cases. •*^ < 

Two other arguments against dual standards are worthy of, serious 
conj^ideration. First, their use heightens the community's sense of 
injustice and, as show n by the CJallup Poll results cited earlier,'^^ the 
community in question includes substantial majorities of all citizens, 
black and white. In light of that fact, recipients of special 
preferences are doubly stigmatized, not onl> as persons of inferior 
abiltty but also as persons who are taking unfair advantages. Worse 
still, because the basis for the special preference is race, the stigma 
inevitably attaches not onl> lo blacks who received it bi]t also lo the 
many blacks who did not and achieved their goals without benefit 
of any special preferences vt-hatsoever. 

The resultant risk of simultaneously reinforcing black feelings of 
inferiority and white feelings of superiority is so great that many 
thoughtful'black people are, appalled and outraged. Kenneth^Clark, 
ihe'black psychologist whose. pioneering research was cited ip the 
famous footnote II to the Brown .decision, put it this way: 'For 
blaqks to be held to lowei'slandards or in some cases no standards at 
all is a most contemptible form of racism."**^ 

The Law School Admission Council has a rather remarkable 
answer for Professor Clark and others like hiTn. In their amicus brief 
in the Bakke case, they argue that there should be no stigma because 
the black applicants admittetl under the separate lower standard are 
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no less qualified than the black and vvhite.applicants admitted under 
» the higher regular standard. The analysis on which their argument is 

based seems to me, as Lbelieve it would to Professor Clark or to any 

other competent psycholiSgisl, virtually incontrovertible. The prpb- 
*^lem is that the conclusion they draw from that analysis is a total 

non sequitur, devoid of logic, fairness, and rationality. . 

Here, in slightly abbreviated form, is their two-part ahaly^is. 

.First, th^y tell us that ugpas and scores on Ihe i saTs allow us to 
■ predict potential law school failures with a very high^degree of ac- 

• curacy 'Mn the lower ranges of the applicant ptioV and that **the use 

* of these predictors for this specific purpose is therefore vita! and 
abandonment would be foolhardy/**'^ Then they go on to the second 
part of their analysis. Let me quote the entire paragraph verbatim: 

Well above the range of probable failure^ however, lies a much larger 
volume of applicants than the schools* total capacity. AJl are fully 
qualified to perform '^^^\\ on law school grades, and many are nearly in,- 
distinguishable on these me|||ires. In this range, where miTsl of the ad- 
missions work must be done, predictions of relative law school KinB 
are Ics^ accurate. But at the same lime, they are less' significant. 
Whether an applicant is predicted for the 40lh oi 50lh percentile of the 
class is a itialler of no real coiiseijiience.^' 

Bravo! A flawless analysis proving that it makes no sense to reject 
competent blacks on the basis of a relative standard so ludicrously 
inflated that it serves only to predict, weakly, "a matter of no real 
consequence.'' a standard that, in its uppcp reaches, is well nigh 
meaningless, lliis is hardly a justificmion for the adoption of dual 
standards. Rather, it is a compelling argument for the abandonment 
of inflated relative standards foi all applicants, white and black/ ft' is 
to that topic that I turn next. 

The Case lor Abandoning Inflated Relative Standards 

There are three main arguments against the u.se j^f inflated relative 
standards. The first is that they bear no meaningful relationship to 
intellectual merit and may be inversely related to important, iionin- 
tellectual traits which are also necessary for competent performance 
in the law as in other fields. The point about the lack of relationship 
'between intellectual merit and tesl scores above the necessary 
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ability level has already been made in the paragraph from^he lsac 
brief quoted above. In Iqss technical language, the point is that test 
score differences above a certain level become, in essence, dif- 
ferences between Tweedledum and Tweediedee. In this realm* the 
maximum, is not necessarily the optimum. More simply still, 
••*more** is. not the same as "better." and the simple-minded 
insistence on regarding it as such is potentially quite harmful and 
counterproductive. 

. This is so, as I tried to explain in an earlier paper.^* because while 
a certain level of v.erbal facility and reasoning ability nlay be 
necessary for competent performance as a lawyer or anything else, 
above that level the differences between superior and hferior 
workers ^rc a function of other characteristics.. Focusing students* 
energies on the meanly competitive task of scrounging about for an 
extra half-point advantage on an exam can be demeaning and de- 
moralizing. It is conducive to ^he development of a fierce but in- 
tellectually contentless competition a^ the expense of intellectual 
depth, originality, or character. Ex hypothesi. it may well serve to 
eliminate from the upper echelons of the'professions many people of 
superior ability who pursue intellectual enlightenment rather than 
points and^as a result, earn scores or grades which, on the average, 
could be predicted to be somewhat lower than those earned by the 
single-mind^dly opportunistic. 

A second argument against the use of inflated relative standards is 
that screening on this basis produces unjustifi^:u feelings of infe- 
riority in hundreds of thousands of people whose abilities are fully 
adequate for (fie positions they aspire to but who are led to believe 
that they have been rejected because their abilities are not good 
enough. Telling them that the standards by which they have been 
judged and found wanting have become so arbitrary as to verge on 
meaninglessness would be more capdid but is unlikely to mollify 
them. Nefther is jt likely to help them undec^tand and deal with the 
late twentieth century worid they live in and th5 special problems it 
has produced, problems which revolve around ifit fact that there are 
too many people and too few places for them. 

The third argumoQt against the use of inflated relarive standards is 

that such an approaclTprovicles the basis for what has become one of 

our most dangerous and least rewarding national pastimes: making 

endless, artificial comparisons of the abilities of whites .versus 

blacks. Please note that 1 am not— repeat, not— challenging the 
« 
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need to make real comparisons on any meaningful basis. If the 
children of the poor are not learning the basic skills necessary 
to give them a real vocational chance and some real vocational 
choices,*we need to know thatfacC so that we can work to change it. 
'Similarly; if the black poor are learning even less than the white 
poor, we need to know that too— again, in order to change it. As my 
colleagues in psychology have repeatedly pointed out, it is as sense- 
less to attack valid tests which reveal genuine deficits as it is to try to 
break "a thermometer because it registers a temperature of 10^.^^^ 
Killing the n\essengfer who brings bad news is just not an effective 
problem-oolving technique. '"^^ ... 

What I am challenging are comparisons that focus on trivial dif- 
ferences which are just that: differences, not deficits requiring 
remedies. Asking whether a greater percentage of whites than of 
blacks obtain lsat scores over 700 is like asking whether more 
whites or more blacks are over seven feet tall. I do not know the 
answer to that question but if I did, I would put it in a trivia collec- 
tion because that is where I think such information belongs 

Instead, such trivia has become a basis for national handwringing, 
weighty policy decisions, and large-scale guilt and inferiority trips 
and is now put forth as a reason for eviscerating a fundamental 
constitutional guarantee. All this, despite the fact that an^applicant, 
black.or white, can have an lsat score literally hundreds of points 
below that and still not have anything that could fairiy be regarded 
as an intellectual deficit for the study of la\v or anything else. The 
purported need for double standards bears little, if any, relationship 
to black deficit^; it is largely a function of inflated relative standards. 
A return to reasonable standards would therefore obviate the need 
for double standards; it is, in essence, as simple as that. 

These, then, are the arguments for the abandonment of inflated 
relative standards. To the best of my knowledge, there are only 
three arguments for the retention of such standards: I) they insure 
constant progress; 2) they maximize competition; and 3) anyway, 
there is no alternative. Regarding the first argument, at this point, it 
should suffice to say that movement per se, up or down, is neither an- 
infallible sign of progress nor an omen of retrogression. The real 
issue here is not progress but prestige: Schools with the highest rela- 
tive standards may not really have better s^^dents and may nbt 
really produce better lawyers but they do have the most prestige, 
and many indivjduals have a heavy investment in that prestige 
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system. StilL it hardly seems worth the^social and constitutional 
cost, especially since there are other, better ways to maintain a 
p/estige system and to keep it related to real rather than artificial 
merit. 

Quality of faculty one time-honored alternative and a good one. 
Thus, if the Yale Law School hrfd abandoned relative standards 
yesterday and equalized the admission chances of every applicant 
with an lsat score of 500 or more, for example, it would still have 
been regarded by many as an especially desirable place to study, if 
for no other reason than because the late Alexander Bickel was 
there.-"^' iVlany admirers of the work of Philip Kurland, myself in- 
cluded, feel the same way about the University of Chicago/' Paul ^ 
Freund of Harvard'^ and Herbert Wechsler of Columbia'** are simi- 
larly regarded by many, and for similarly good reasons. 

The second argument for relative standards is that they maximize 
competition and, while it is certainly clear that they do that, it is 
much less clear whether that is really an absolute good, in and of it- 
self. John Dean would probably get very high marks on a scale of 
sheer^competitiveness,'but that seems to be at least as much a part - 
of the problem as it is of the solution. 

Perhaps because the foregoing arguments are so weak, most , 
amicus briefs in the Bakkc case which defended the current relative 
dual-standards system did so by relying almost entirely on the third 
and last argument for it: There's no alternative. a 



III. An Alternative: Building 
on Mayor Washington's Foundation 

Since I have claimed, ♦hroughout this presentation, that there is a 
better alternative, I had best use whJvJ little time remains to at least 
sketch in the main contours of that afternative. What I am recom- 
mending is that American universities start afresh by abandoning 
relative standards and dual standards and returning to a single, 
absolute, necelsary-ability-level st.mdarti for ev^ ryone. That was 
the approach adopted by V/alter Washington, the black mayor of the 
District of Co^imbia, and his police chief, Jerry V. Wilson, to screen 
applicants for jobs as police officers. 

They, too, used a test of verbal and reasoning ability. Test 21 , but 
instead of encouraging endless, senseless, spiralling competition. 
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they used a single, absolute cut-off score chosen to reflect thejevel 
of ability necessary for competent performance as a police officer. * 
They then accepted all applicants who scored above that line and 
rejected all applicants who scored below it. By combining this 
reasonable screening metf\od with an affirmative-action recruitment 
program, they were able to hire large numbers of competent police 
officers. No double standards and no racial quotas were used and 
none were needed: In the initial hiring period, 44 percent of all new 
officers hired were black. ''^ Later, that figure rose to 57 percent.''^ 

This^ was the. screening system challenged in the Washington v, 
Davis case and upheld by the Sup^^eme Court as consistent with the 
Fourteenth Amendment. It- is the system 1 believe American 
Universities should also adopt and build upon. OfR):ials of many such 
institutions will immediately protest: 'impossible! We can't accept 
all qualified applicants: there just aren't enough openings. 'There is 
a simple answer jo that: Use current tests like the 1-sat to separate 
the qualitied from the unqualified and then equalize the chances of 
every qualified applicant by selecting from among them on a random 
basis. . 

Randomization has a great many advantages over the present 
univecsity selection liystem. I will very quickly list the niore im- 
portJ^nt ones and then conclude this presentation with some sugges- 
tions about hpw randomization can bo coupled with other methods 
in ways that enable us to build on the Washington foundation in 
order to reduce our ignorance and increase the efficiency and fair- 
ness of our screertiiig systems, t 

The main advantages of randomization are the.e: First, it would 
maximize diversity, not just between racial groups but within them, 
and without using qubtas and the insoluble problems inherent in 
their use in a society as heterogeneous as ours. Second, it would 
reduce the community's sense of injustice: No group could be ac- 
cused of having an unfair advantage over any other groirp. Third, it 
would curtail the spread of unwarranted feelings of inferiority: The 
luck of the draw rather than lack of ability would become an ac- 
knowledged basis for many rejections. Fourth, if would reduce 
destructive and pointless competition and hostility between indi- 
vidual^ and groups. Fifth, and perhaps ^most important, it would 
,focus.everyone*s attention on the leal problem; the fact that we have 
too many people and too few places. 

Why that is so and wh^l can and should be done about it i^ a topic 
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worthy of extended discussion in its own right, but one that is be- 
^yond the scope of this paper. Here, the focus is on dealing with the 
probleni as it currently exists by improving external screening 
methodr Wiser use of the many, good tests of intellectual abilities 
we alrdady possess will solve some of our immediate problems. 
Further progress, however, depends iq^on our learning how to ^ 
identify and measure other relevant abilities so that instead of 
screening for ortly one necessary ability or quality, we can also 
screen for other, equally important ones. To' do that, we must be 
able fo test new measures of additional qualities, discarding those 
that prove invalid and r'^taining those that prove valid. 

American universiti^i are already experimenting with a variety of 
additional screening ideas, using subjective factors to select from 
among an artificially narrowed group of extremely high scorers on 
current measures. Such experimentation is inevitable; it cannot and 
should not be elimiitaited but it could and Should be done in a more 
systematic, scientific fashion so that it will teach us more than it cur- 
rently does. \ ' c^ . > " 

If, instead of using the same un validated, second-stage screening 
methods on all qualified applicants, universities were to select at 
least half of them at random and the other half on the basis of 
whatever new fadtors or measures seem promising tojhem, we 
would be in a better position to learn from our mistakes and in less 
danger of repeating them. This happier outcome would be especially 
likely if universities were to then commission careful, long-term 
follow-up studies of how the members ^ each selection group 
perform, not only in school but afterwards as well. Because ran- 
domization would largely obviate the restrictidn-of-range problem 
which has dogged researchers in this area,, the chances of success- 
. fully identifying valid new predictors would be markedly improved. 

If such procedures are followed, we should, eventually, learn how 
to screen in ways that genuinely foster and reward real merit for all 
of our citizens. The goal, in sum, is to get all of us off the trivia 
treadmill, redirecting our energies into the struggle to achieve a true 
meritocracy for the benefit of all of our citizens. 
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program. The median LSAT score of those minority students was, 683 in 
1971, sitause LSAT scores have tended to rise each year for the past 
several years, the odds are great that current medians are higher still, 
and that the overall mean-median discrepancy is even larger. 

10. Evans Report at 7. See also Report of Minority Groups Project in 
American Association of Law Schools Proceedings 172 (1965). In the 
latter source, blacks were estimated to constitute only about 1.3 perce nt 
of all law students in 1964 (roughly 700 out of 54.265)'. 

1 1. Sec, SynipoMiini. DisadvantaiieJ Students and Le^al Edm atUm — 
Prof^rams for Affirmative Aetion. 19,70 fOL. L. REV. 227. Sec also 
Brief Amicus Curiae for the Association of American Law Schools in 
the Bakke ^*ase at 24. VPhus, by the mid I07()s. in virtually all schools, 
in one way of another, ft preference inlhe application of admission stan- 
dards was in fact afforded to applicants from minority groups.'' 

12. 5(r The New York Times. Sunday. Ma> I. 1977. p. A. 33, col. 1. 

13. /(/. 

• 

14. See AnnicUN Curiae Brief of the National Conference of Black Lawyers 
in the Bakkc case for a detailed examination of som<; of* the grosser 
inadequacies in the data provided by admissions officers of the 
University of California Medical School at Davis on criteria used to 
admit students through their regulai and special admissions programs. 
One need not accept the conclusion of the brief's authors— tjiat the 
university\s position is not really adverse to BakkeS— to share their 
concern over what appears lo be lack of clarity and/or candor about 
how criteria wpre'Jefined and applied. Such concein seems well war- 
ranted in light of available evidence on the invalidity of subjective 
methods of evaluation. See Linn & Winograd. Nen York University Ad-^ 
missions interview Stndw Report #LSAC-69-2, Law School Admission 



22 



32 



barbara Lerner 



' Council, REPORTS OF LSAC SPONSORED RESEARCH: VOL. I. 
J 949-69. 

15. 5ec, e.g.. Pin.skcr v. Pacific Coast Society of Orthodontists. 12 Cal. 3d 
541 fl974) for a recent statement tnat due process requires admission 
standards to be reasonably certain and consistently applied, 

16. The same point has been made clearly and forcefully in several other 
recent articles. See. ^.g., L»vinsky; Dc Funis v. Ode^^aard: The Nan- 

, Decision with a Message^ 1975 COLUM. L. REV. 520. 

17. See EEOC Standard Race/Ethnic Categories. 41 Fed. Reg. 17601-02. 
(April 27, 1976), 

18. See De Funis v. Odegaard. 416 U.S. at 338^whcre Mr. Justice Douglas, 
dissenting, noted that the speci;\,l admissfons program at,the University, 
of Washington Law School "indudcd Filipinos, but excluded Chinese 

' and Japanese." Japanese-Americans were al,so eliminated from the spe- 
cial admissions program at the University of CaHft)rnia Law School, 
when the faculty found that members of that group were being admitted 
in substantial numbers through the regular admissions* program. See 
Report on Special Admissions at Boalt Hall After Bakke. 28 J. LEGAL 
ED. 363(1977). 

19. COUNTING THE FORGOTTEN. U.S. Commission on Civil Rights 43 
(1974). 

20. For an accoun! of the largely unsuccessful ctTorfs of n on- Hispanic white 
ethnic Americans to get appropriate governmental agencies to compile 
current statistics on how members of their groups are faring in contem- 
porary society. Brief Amicus Curiae of ihe Polish American 
Congress ^t aL in the Bakke case. See also Hai^lin. The Goals of In- 
tegration, in PARSONS & CLARK. THE NEGRO 

(1967): "[Tlhe limitations of the census categories which recognize 
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(at 699). 

^ Although the Evans RcportA\\so tries to lump all iion- Hispanic w hites 
together, some tip-of-the-iceberg indications of hepeiogeneity within 
that wastcbaskel category can be gleaned by examining the data pro- 
vided in ^ppendi.x E on LSA T scores of Jhree groups which are 
probably all or mainly white! those who idemified themselves as white 
(48.249 applicants), those who refused to identify themselves (18,745 
applicants), and those who identified themselves as "other*" (2.l.'>2 ap- 
plicants). Calculations based on these datii show that 3."^ percent of the 
"whiles" achieved LSAT scores ^ 600. but 4,0\percent of the 
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unidcntifieds and only 24 percent of th&<;others'' did likewise. With 
regard to the-reported discrepancies between black and "white' LSAT 
scores, it is also important to note that a larger percentage of black than 
of white college graduates apply to law schools. The result is that a less 
selected group of black applicants^ competing with a more selected 
group of white applicants with a consequent exaggeration of the mag- 
nitude of intergroup differences. 

More generally, historical data on the differential schooj achievement 
of various white ethnic groups are summarized in Greer, Immim^nts 
• and School Performance, in THE GREAT SCHOOL LEGEND ( m 
and in RAVITCH. THE GREAT SCHOOL WARS (1974). For more 
recent data on the persistence'of ethnic grot^p differences, see Gross, 
Learning Readiness in Two Jc\vish Groups, Centef for Urban Educa- 
tion (1967): Lesser, Fifer & Clarke, Mental Abilities of Children from 
Different Social Classes and Cidtural, Groups, Monographs of the So- 
ciety for* Research in Child Developm'^m, Vol. 30, No. 4 (1965): 
Miyoribanks, Ethnic antt-EfiVironnu nUd Influt m cs on Mental Abilities, 
J8 AM. J. SOCIOL. 323 ( 1972); Sch Aartz, The Culturally Advantaged: 
A Study of Japancse-Amcriban Fuoih. in EPPS, ed., RACE RELA- 
TIONS (1973). 

21. ABA Uw' Schools and Bar AdpiisMon Requirements: A Review of 
Legal Education in the United States, 42, ( 1976). 

22. Report of Minority Group Project in AALS PROCEEDINGS 172 
(1965). 

23. Evans Report ?Xl . 

24. Atelsek & Gomberg, Ba^ helors Degrees Aw arded to Minority Students, 
1973-74, American Council on Education. Higher EduciUion Panel 
Report No. 24 (1977). 

25. Minority Group Pattiiipation in Grdiluate lldmation. National Board 
on Graduate Education ( 1976). 

26. See Report on Survey of 1971-72 CLHO Graduates in Amicus Curiae 
Brief of the Council on Legal Education Opportunity in the Bakke case, 
41-42. 

27. /J. at 45. 

28. See Chicago Daily News. Saturday-Sunday. August 6-7. 1977. p. 4. 

29 Bureau of the Census. Current Population Reports. Series P-60, No. 
103. Money Income and Poverty Status of Families and Persons m the 
United States: 1975 and 1974 Revisions (Advance Report. 1976). 

30. Evans Report at 59. 
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^Ernest M. Bernal Jr. 

^Associate Professor ofBic^ural Bilingual Studies 
The University oftexq^ft San Antonio 

I find myself in a most peculiar position: While charmed by ourl 
speaker's pulsar presentation, I am compelled by my difference in 
perspective to don my black helmet, visor, and body armor and' 
engage in some Star Wars, even at the risk of making her appear to 
be Princess Leia. Admittedly, Lerner wishes to expand her paper 
and'^isiookmg for a critique.- I urge iier to adjust her deflector 
shields, check the atomic batteries in her light sabre, and insert'a 
freshVefill in her ballpoint pen. 

To/begin ^i(h, while I don't wish to dispute Dr. Barbara Lerner*s 
personal claim to hold in high regard the ethnic diversity which 
she herself has experienced. I must, say that her presentation 
demonstrates an almost smgular insensitivity to cultural pluralism. 
Her credibility is shaken when she enumerates no less than nine 
Anglo minorities but insists on recognizing only four other minority 
groups under the rubric of their being "favored." Instead of seeing 
the cultural differences which exist within these four broad 
classifications, however, she promptly reduces them to two-and-a- 
half, and thereafter refers to these collectively as "blacks," ap- 
parently out of her concern for efficiency in communication. I would 
hope that our speaker, as a member of one of the minorities whi(;Ji ' 
constitute the privileged and educationally advantaged majority, 
would exhibit both noblesse oblige and ethnographic accuracy. Cer- 
tainly no person tonscious and proud of her or his minority culture 
appreciates being reduced to the status of shorthand. 

It might be instructive for her to stop regardin*g us all as mere 
ethnics and to start thinking of us as ethnics who are either dominant 
or non-dominant, to consider whether a group's historical, cultural, 
or linguistic background either gives it access to the formal organiza- 
tions or institutions of our sodety or becomes an impediment to its 
members' participation. In this way, she would not be tempted to 
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confuse numbers with proportions when discussing poverty and 
> * would g^n an opposite perspective on why so many whites from 

families with yearly incomes under S6,500 apply to law schools. ^ 

I have long felt that the term admission stapdards has been an un- 
fortunate one. It seems to imply the very rat race which Lerner de- 
cries. Requirements or prerequisites would perhaps be more to th^ 
point, leaving standards to the faculties of ourjprofessional schools 
and the members of our^icensing^ boards, 4o the competencies 
necessary to p;/ss courses, graduate; and be certified. A school may 
insist on prerequisite skills or achievements, but its standards 
should not be confused with applicants' characteristics. 

Our speaker assumes that admissions scores are interpretable in 
the same way for members of both dominant and nondominant 
ethnic groups. Some psychologists would generally favor this posi- 
tion (2); others (I) would not. 1 suspect that there are irrelevant fac- 
tors in admissions tests for nondominant ethnic applicants which ^io 
not obtain for their dominant efhnic competitoVs. While this issue is 
subject to scientific resolution and explanation, so far. little has 
been done to investigate it directly. 

Next, as intuitively attractive as Lernefsjiolution seems at first— 
to select randomly, from among all those who qualify— I don't 
believe that it is practicable, at least not in the immediate future. 
For however inadequate the , screening instruments may be by 
themselves or however low. empirically determined cutoff scores 
could be set, 1 predict that all professional schools will continue to 
use these tests with undue weight and that the prestigious ones will 
insist on higher marks than necessary. Given the glut of applicants 
and a certain desire to save face, would lesser institutions be far be- 
hind in their admissions requirements? Also, random selection 
would encourage students to apply to virtually every professional 
school in the country, thereby creating problems for institutions and 
for students, especially those from poverty backgrounds, who too 
' frequently can attend only those schools which are located nearby. 

ETS has recently released m study by Evans on the admissions 
practices in the nation's law schools. Lerner^s data on enrollment 
• trends and her contention that there exists a dual system of ad- 
missions are supported by this report. According to an article about 
the Evans study published in ets Developments (4). --The central 
question addressed . . . was: What would happen if law school ad- 
missions committees were forced to disregard racial factors in mak- 
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ing admissions decisions?" Assuming the continuation of Lerncr's 
''inflated relative stancjards'* by law schools, one analysis of the 
question, as report. i in the ets publication, estimates that ''the 
decrease in the number of blacks accepted would-be 60 percent and 
the number ofChicanos 40 percent/' Only I percent of black appli- 
cants (10 persons> and 4 percent'of Chicano applicants ( 1 1 persons) . 
would be admitte^d ahnuially id the "most seJective" law schools. 
Most blacks and .Chicanbs--73 percent and 57 percent, respec- 
tively— would be Toand in jhe 'least selective-' institutions, 
whereas only 27 percent of the' '* White and Upfdentified" category 
Would be so situated. If, as Lerner intimates, we were to consider 
economic criteria instead of rate as one basis of affirmative action in 
admissions practices, the Evans report concludes that this "would 
not result in the admission of substantial numbers of minority 
students because the vast n4i\jority of low-income candidates is 
white.'* 

So the conclusion seems inescap-^-4 that, constitutional or Itgal 
or not, the dual system of admissions has provided access for 
significant numbers^ (albeit still not enough) of nondominant ethnic 
Aippiicants to law schools . . . and to medical and graduate scljools 
as well, including the selective ones. 

A Look at the Past ' 

I believe it is important to read into the record of this conference a 
cursory historical account of the struggle to enh ance the access of 
minorities to graduate and professional schools, an account which 
our speaker should consider very deliberately. , 

Dr. Lerner points out that in the 1950s, the number of applications 
to professional school's necessitated some form of external screen- 
ing. The fifties also saw a significant rise in minority enrollments, 
both in undcrgracfaate and in professional schools. Educators in,- 
yolved in equal educational opportunity in higher education have 
long argued that admissions tests and practices serve not only to 
keep the number of minority entrants low but also to^ relegate them, 
to schools with lower standards. The data suggest that this has, in 
fact, happenef . It really should not be surprising, then, lo find, as 
Lerner did, that on the average, minority professional school 
graduates are earning less than their Jominant-group counterparts. 
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^Still, a number of us ***blacks^^ persisted through the fifties and 
found our way into graduate and professional schools during the six- 
ties, in the meantime, our burgeoning minority populations kept en- 
tering undergraduate schools in even greater • numbers and 
percentages. The need to measure the ''other relevant abilities'' was 
recognized by us even then, because those -of us who gained ad- 
mission to professional schools (usually after a venture into tbe 
world) were aware, of how many of our talented, minority ethnic ac- 
quaintances were being excluded, not because they weren't 
competent but because they didn't have the requisite scores. To 
have one's gifts and talents go undetected is an even deeper frustra- 
tion than that which LCrner describes. 

Many of us have worked hard since those days to ensure that suc- 
ceeding college graduates have a greater opportunity to attend 
professional schools than our own peers , did, and we have made 
some gains. We have striven within the field of testing— as yet 
unsuccessfully— to have measures of these ''other relevant varia- 
bles'' developed and to study ethnic differences in admissions test 
scores; we'v£ been in the courts; weVe organized and bargained 
politically; and we've taken the fight into the streets when neces- 
sary ... and we may again. Only in recent years, when admissions 
denials .have>a-eached enimrrassing proportions, have we managed 
to win a few slots here and there in professional schools. And in rare 
instances, professional schools have sought us out, have used their 
own initiative to set aside a few places in order to secure our partici- 
pation. 

I have reviewed these historical developments so that you may 
understa^id that all these efforts are not, as Lerner somewhat 
msouciantly suggests, designed t^keep our college graduates out of 
poverty, but to give our competent people the options which are 
necessary for indfvidual fulfillment and ethnic viability. , 

Our speaker may argue— and validly so— that these measures 
have not corrected the basic problems which she has addressed. 
Nevertheless, we have achieved at least a partial redress, a partial 
implementation. What is important is that even a small measure of 
moral justice will not be relinquished to satisfy the Constitution 
when the result will likely be moral injustice. Lerner has devoted 
much energy to making her solution compatible with the Constitu- 
tion. Even now there are developments in the Bakke case whidh sug- 
gest that It may be decided on other-than-constitutional grounds (3). 

32 

9^ 41 



Ernest M.BemaUr 



Perhaps our speaker should focus her efforts on demonstrating that 
her solution is ethaically equitable as well. ^ 

Finally, we can live for a while with the discomfort ofThpwing 
that preferential admissions will offend many of our .citizens, 
members of dominant and nondominant groups alike. Dr. Barbara 
Lerner has gotta know that, like Darth Vader of Star Wars) we 
**blacks," we can hang tough! / 

But we ar'e also profoundly human. So human, in fact, riiat we, 
too, can be in touch with the Light Side of the Force. / 
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I find the statement of Dr! Lerner's case against dual standards to be 
persuasive probably because I've argued the same case myself man/ 
times in legal briefs and in more informal arguments. I also agree 
with the assessment of the enormous constitutional cost inherent in 
any standard based on race. So let me focusi my discussion on Dr. 
Lerner's solution. 

I find the statement of the case nn favor of a minimum score 
reflecting competency accpmpanied by randomization to have 
several fundamental flaws. I should point out first that these flaws 
are primarily political and philosophical. This is a solution that, if 
implemented' as Dr. Lerner intends, certainly would withstand, 
constitutional scrutiny and would reduce legal risks substantially.* 
But reducing risks in the courts also involves costs that I think are 
substantial. Let me try to outline four significant problems. 

First, the fundamer.ta! ;w^;»jmption of this approach is that such a * 
cut-off score can be set accurately enough to support random selec- 
tion. If it can't, you simply invitt a whole new round of criticism for 
unjust treatment. The higher the minimum score is, of course, the 
greater the chance high-scoring candidates will have of being chosen 
under the random-selection system. If the score is set too low, high- 
scoring candidates will believe it is unfair to group them in a large 
category of people which includes many low-scoring candidates be- 
cause that reduces their chance of being ^elected. If the score is set 
too high, low-scoring candidates will have the same aigumcnts they 
have under the current system. J think the probability that »» 
minimum score can be set accurately enough to support random se- 
lection is beguilingly overstated here. 

Second, even if a minimum score could be set m a satisfactory 
manner, this approach assumes that the tests are useful only to 
predict failuie and not to predict relative degrees of success or to 
maximize success. Several points ought to be made in that regard. 
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It's conceded that test scores, correlate with performance in 
professional school, and the evidence is not at all conclusive that 
distinctions at the upper end of the scale are not meaningful. Even if 
fine-tuned distinctions among individuals are not appropriate, this 
does not mean either that the hijgher performance predicted by 
higher test scores is irrelevant or'that the use of test scores to maxi* 
mize group performance is irrational. The papet points out that the 
score levels now used by law schools for external screening are 
much too high forjudging competency, and the phrases ^'grossly 
inflated standards" 3nd 'ludicrously inflated standards" are used 
to describe these score levels. J*m concerned whether there is 
sufficient evidence that these standards are grossly or ludicrously 
inflated if relative success rather than probable competence is being 
predicted. It seems at least politically if not intellectually unsatisfac- 
tory to lock at minimal scores reflecting competence in a graduate or 
professional school where enormously valuable and very scarce 
resources are being used. The use of these resources on candidates 
who have been selected because they probably won't fail rathe- than 
because they will maximize the contribution to be made to the in- 
tellectual endeavor seems unlikely to raise enthusiasm in the hearts 
of the taxpayers or the alumni or anyone else who is contributing to 
tjje system. In short, you have to consider whether the public will 
stand for such a solution. After six monthsrin political office, my in- 
stinct is that they won't. 

Absent a showing that increased skills do not serve professional 
competence or that increased skills are indeed negatively correlated 
with other important character traits, it is entirely appropriate for 
law schools to try to select the best qualified students by relying in 
pari on test scores to identify those students. In this connection, it 
seems to me that Lerner's paper confuses the nature of test scores 
and grades— or at least the results of these two systems. It is not 
meanmgful to describe students as narrowly trying to eke out an ad- 
ditional few points on standardi^Ted tests. There is no evidence that 
tests have an impact on the nature or degree of competition aniong 
students. It is the presence of grades thai/stimulr.tes competition and 
determines almost entirely the nature of the competition. Moreoveii 
tests play a useful Vole in correlating for discrepancies in the subjec- 
tive judgments underlying some grades. 

Third, educational institution^) have an important interest in 
avoiding the lowest-common-denominator approach whether that 
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denominator is phrased in terms of "full competency'* or some 
other classifier. Selecting the most qualified as opposed to the at 
least minimally qualified students, particularly in a professional 
school, contributes to the quality of the education offered by the in- 
stitution. In many situations,, the skills of the students affect the 
pace and tha quality of the discussion in the classrooln. This is much 
more than a prestige factor, as described in DpLerner's paper. 

Fourth,, randomization may be psychometrically pure but it runs 
strongly against the tradition of merit-based selection which^ has 
been and remains a basic fenet of American society. This' tradition 
recognizes the* element of gamesmanship in the competition^ for 
grades and scores. It recognizes that the A student who gets a 720 on 
the tsAT may. not become a better lawyer than the A- student who 
gets a 690 on the lsat. Nohettjeless, it's prepared to declare the A 
student with a 720 score the winner in a competition for a slot at a 
preferred law s^hoal because of the belief thslt the element of merit 
is being measyred to the extent possible. Only a most determined 
demonstration that there is no element of merit involved would 
persuade the public to the contrary, ^ 

Randomization also has other drawbacks. Randomization may 
minimize the feelings of rejection of those who are on the lov/ end of 
tfie score totem pole, but you have to take into consideration that it 
will maximize the feelings of frustration and demoralization of those 
who are on the high end of that totem pole. On the "feelings scale/' 
if you introduce a system of randomization, you may wind up with a 
draw. 



A Better Solution 

In sum, I think a better solution is perhaps the one that Win Manning 
has, discussed extensively in publications and elsewhere: Schools 
should set minimum standards for acceptable performance in terms 
of test scores alonj^* and notify candidates that they have the basic 
ability to succeed in professional school. Then, at a second stage, 
they should use grades, test scores, scored interviews, scored back- 
ground questionnaires — whatever means are available — to select 
from within this competent group those most likely to maximize the 
contribution to the intellectual endeavor of a graduate school or 
professional school and beyond that in the profession itself. In so 
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doing, the chance is minimized that the selection/system will un- 
dermine public confidence in educational institutions and in the wise 
use of an, enormous amount of the public's money and energy in 
those institutions. 
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My assigyiment to speak on the influence of the law on professional 
measurement standards is a difficult one, as the amount of material 
that a' measurement specialist must master in order to discuss this 
issue intelligently is awesome. The relevant law must be inferred 
from a broad corpus of statutes directed primarily to more general 
issues^ from multiple administrative guidelines issued by several 
federal executive agen ies, from conflicting case law. arid from a 
plethora of ''definitive* but contradictory legal analyses. More has 
been written relevant to this issue in the last five years than most 
nonlawyers could possibly read and assimilate on a part-time basis. 
Yet, as should become clear as this paper develops, those concerned ' 
with educational and i)sychological testing today cannot do' their 
work without knowing how the law will affect their professional 
practice. Those of us dedicated to the advancement of the testing 
profession will need tolproceed slowly to try to make some coherent 
sense out of the legal confusion before us by working closely with 
legal counsel and then to begin to make such changes in our 
professional measurement standards and practices as law and cir- 
cumstances demand. As concerned professionals, we m-^y also 
decide that it is our responsibility to play an active role in the 
development of pertinent law. 

In my presentation today. I shall not make any predictions as to 
how various issues will be resolved or when they will be resolved. 
Rather I shall attempt to identify those activities that we ought to un- 
dertake now regardless of how these issues may be settled in the 
courts or elsewhere. I shall touch on four specific issues and one 
general issue. First, I shall discuss the question of funding allocation 
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uhd^r Title I of the Elementary and Secondary Education Act (esea) 
of'l965 and indicate how the current discussiQn about poverty-level 
versus ability-level' funding could affect measurement standards. 
Then I shall turn to the topic of reverse discrimination as this issue 
has developed in the context ofBnkkey Regents of the University of 
California, a case on which oral argument was heard before the 
Supreme Court on October 12, 1977 and on which a ruling can be ex- 
pected this term. Third, I shall discuss the question of federal 
guidelines for employment selection noting that the first uniform 
guidelines are expected to be released shortly. Then I shall turn to 
the topic of the reporting of testing irregularities and a discussion of 
the questions it raises about the relationship between the educa- 
tional institution and the student as this relationship is mediated by 
the testing organization. I will refer to some currently proposed 
legislation to regulate the testijig industry tHR 6776, the Harrington 
bill). Finally V I will discussal t )pic that is central to each of the pre- 
vious four specific issues: Wl o is responsible and accountable for 
, educational decision ^^^^^^ 



Funding: Povertyivs. Ability 

Current funding to school districts under Title I is based on a compli- 
cated measure of poverty. The assumption has been that poverty 
level is a variable that can be measured reasonably well and one that 
is highly correlated with educational disadvantage. By educational 
disadvantage I mean the net effect of those characteristics of a 
student's environment that provide less than normal exposure to 
factors that motivate and facilitate educational growth. Such charac- 
teristics affecting educational disadvantage may include, but cer- 
tainly are not li^n^Ied to, the following: educational level of the 
parents, siblings, and peer group; physical facilities in the home; 
economic status of the family; acculturation of the parents, siblings, 
and peer group; language proficiency of the parents, siblings^ and 
peer groirp; quality of formal schcfoling to date; and possibly, 
number and spacing of siblings. As a measure of educational disad- 
vantage, poverty level alone has many deficiencies. More complex 
measures based on a combination of the variables mentioned above 
.will Ije difficult to fashion, but should in the end prove more valid. 

■ . 
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In the 1974 amendment* to the Elementary and Secondary Educa- 
tion Act of 1%5 mandating the current evaluation of Title I, a 
new definition of educational disadvantage was introduced for the 
purpose of evaluation, though no change was made in the formula 
for funding allocation, which remained entirely a function of poverty 
level. The new definition reads as ^llows; 

... the term "educationally disadvantaged children" refers to children 
who are achieving one or more years behind the achievement expected* 
ai the r.ppropriaie grade level for such children. 

This definition would seem either to assume that low attainment ac- 
curately indicates the negative influpnce of the factors mentioned 
above or, alternatively, to assume that the function of Title I legisla- 
tion is not Ito ensure equal educational opportunity but rather to 
guarantee equal educational attainment. Neither of these assump- 
tions seems valid and, as a result, I think it may be very difficult to 
present results of the Title I evaluation in an unambiguous way. 
Professor William Coffman, Director of the Iowa Testing Programs, 
^has pointed out to me that with the Title I evaluation definition of 
dlsa(hantagi\ the percentage of disadvantaged students in the na- 
tional norm group for the Mathematics section of the Iowa Tests of 
Basic Skills rises from 14.'5 to 34.25 from the third to the eighth 
grade. It is not that more students are becoming disadvantaged; only 
that grade-equivalent scores are being used inappropriately. 

The problem becomes more^cute with the introduction of HR 
7571 (the Quie amendment), a bill that proposes to change the basis 
of Title I funding to a formula based on ability level. That is to say 
that Title I money would flow to districts according to the number 
of students whose test scores revealed them to be, to a specified 
degree, below the standard for their age. It must be granted that 
poverty level is not an iOeaTSurrogate for educational disadvantage, 
and the quantification and scalmg of the variables I have mentioned 
has yet to be done. Yet it might be hoped that a niodificatior of one 
of these approaches might be ijound that would be^ preferable to the 
proposed ability-levelTunding. One reason for seeking an alternative 
approach has bern given. Another devastating one follows. V 

When an educational test is administered, it is assumed that the 
teachers have motivated their students to do well. When working 
with disadvantaged minorit> students, this assumption sometimes is 
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teniious, and this is sometimes viewed as a very serious problem. 
But consider what, hypothetically, would happen if district funding 
for first grade and beyond depended on students attaining low test 
scores at age seven. Such a system lends unmistakable incentive for 
meniets of the teaching profession, members of local school 
boards, and taxpayers to arrange for low test scores at this level. 
This could be accomplished by withholding compensatory treatment 
at the preschool and kindergarten levels (when it is most effective), 
' by failing to motivate students.to ptrform,well on the tests, and by a 
variety of other strategems. (For d more detailed treatment of this 
topic, I refer you to the Feldmesse^ report [4].) If funding by ability 
level comes into being, we shall need some new measurement stan- 
dards to help avoid this danger, but it is questionable whether any 
standards could effectively solve so fundamental a problem. 

What, then, is the best available basis for funding? Because of the 
well-recognized difficulties of poverty-level funding, which have 
' been well documented, the lack of a sound psychometric sqahng of 
educational disadvantage, and the above criticisms of ability-level 
funding, I would recommend an interim compromise. First, let me 
say that I think the idea of using the National Assessment of Educa- 
tional Progress as the basis for census survey data for allocating 
funds, as studied by Harnischfeger. Huckins. and Wiley (5). is an 
interesting one. My primary quarrel is with the variable that »'aey 
were directed to study. What I would re'commend is funding on the 
basis of the distribution of mothers" educational level in each dis- 
trict. Evidence in support of the belief that this is the best available 
single measure of educational disadvantage is by no means solid; 
however, further study might support this conjecture. What is im- 
portant is that mothers" educational level would probably do the re- 
quired job very effectively with only negligible bothersome side ef- 
fects and it would be a move in the right direction pending further 
psychometric development. Certainly the problem of preparing 
professional measurement standards would be easier if this interim 
approach were adopted. 

The BakKe Case: Measuring Individual Disadvantage 

There is no need to review the background of B(Mc »• Rcf-cnts of 
the University of Colifontia. This has .been done '^ The New York 
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*Timfe5, in. the Des Moines Register, and in the lowo City Press 
Citizen almost every da>| for the past two months. What is interest-^ 

^ing to me Is the position. taken by the Justice Department and even 
more so, the position almost taken. While the Department did come 
out in support, of the University of California at Davis and against 
Bakke, in support of raU as an indicator of disadvantage, and irv 
support of goals for affirmative action, the Department stoppeU 
short of supporting racial and ethnic quotas. In fact, an early draft of 
the brief reportedly conceded that quotas are probably unconstitu- 
tional. • j 

Just how hard the Supreme Court will come down oa quotas is 
hard to say. Whether they will choose this as the moment to codify 
the evolution in American political thought away from concepts of 
group parity and ethnic quotas is uncertain. But that evolution is^a 
reality with or without codification. What this means is that ways 
other than the use of ethnic or racial quotas must be developed to 
help eliminate the wide! variability in educational advantage now be- 
ing experienced by young Americans. The only way that I can set 
this happening is through the quantification and scaling of a measure 
of individual disadvantage with race or ethnicity possibly being used 
on an interim basis as one'of several components of that measure, 
with the allocation of funding, individual attention, and, in some cir- 
cumstances, selection being based, in part, on that measure. This 
same principle can be applied at 'first-grade level, as indicated pre- 
viously, and at the professional-school level. Given two applicants 
with identical Law School Admission Test scores, for example, it 
should be 'possible to measure the relative educational advantage 
each student has enjoyed to that point and to take this into account 
in the decision-making process, not as a predictor variable but as a 
component of value or utility. Those who attain a particular stan- 
dard despite a disadvantaged background are, for a variety of 
reasons, to be preferred to those from a more advantaged back- 
ground who attain the same standard. In many cases, such a policv 
should involve not only preferential selection but a\^o special prepa- 
ratory treatment before entry into law school. Rreparatory schools 
at all levels, that traditional luxury of weahhy families, ought to be 
made available, at federal expense, to promising disadvantaged 
students. 

Should the Supreme Court support Bakke in his claim that racial 
and ethnic quotas are unconstitutional, we shall need a highly ac- 
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celerated developmental effort for the measurement of* disad- 
vantage. We shall need to identify and study all the variables we can 
find, including racial identification, that relate to disadvantage in a 
causal way and create one or more measures of disadvantage appro- 
priate for various applications. If the Court fails to rule on the issue, 
we shall have more time, but the job will still need to be done. Ma- 
jority tolerance of preferential admission based on racial quotas will 
not last long nor will minority tolerance of unremedied educational 
disadvantage. Certainly, variability in educational disadvantage will 
not disappear overnight, and so we shaU indeed need to measure it. 

Should the Court choose to take an extreme position and affirm 
the constitutionality of group-pUrity concepts arid racial quotas, the 
problems we face will be more difficult. As my colleague, Dorsey D* 
Ellis Jr., and,! (7) argued in the May 1977 issue of the American 
Psychologist, there is no statistically or legally. coherent formulation 
of the concept of group parity and no present standard for racial 
identification. To implement a policy of racial preference in any 
scientific or legally acceptable way would require, for example, a 
definition of black Americqn and a measurtfmcnt procedure for pro- 
viding racial classifications. To use what for me is offensive termi- 
nology, we would, in a reenjiptmcnt of Plcssy v Ferguson ( 1896). be 
compelled to stipulate what percentage of blackness, say. is 
necessary for preferential classification, and then we would be com- 
pelled to measure perjccntage of blackness for every person seeking 
preferential treatment. 

Question: Is a person with three white grandparents and one black 
grandparent himself black or white? What ^ibout one black great- 
grandparent and seven white? .And. indeed, wtr*- the grandparents 
themselves black or white? 

*The support of group parity and racial quotas by the Supreme 
Court would cenainly raise this ditticult and distasteful measure- 
ment problem. Just that .sort of activity is now central to tht opera- 
tion of the Bureau of Indian Affairs, with little benefit to native 
Americans (see 7). 

Let me summarize my position on this issue. To claim that we 
should be color-blind in academic admissions in contemporary 
American society is visionary but impractical; to claim that we 
should have racial quotas to eliminate educational di-sadvantage is to 
be ingenuously blinded by color. 
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Codifying Fadaral Guidelines * 

• The history of federal guidelines for ernployment selection is long 
and'complicated. I shall touch only on recent highlights. In 1970, the 
Equal Employment Opportunity Commission isstied stringera 
guidelines tiiat fought to prevent discrimination against minorities. 
On November 23. 1976, Federal Executive Af^e^cy Guidelines wjre 
issued jointly by the Justice and Labor Departments j^nd the Civil 
Service Commission. These^uidelines were judged to be consistent 
with the test standards published jointly by the American 
Psychological Association, the American Educational Research 
Association, and the National Council on Measurement in Educa- 
tion and received much professional !>»jpport as being.a step in the 
right direction, it being felt that the earlier EEOC Guidelines were 

>not consistent with contemporary personnel psychology. 

I shall not comment on this topic at any length for two reasons. 
First, new uniform guidelines are anticipated shortly. Second, my 
discussion ofBakke has explored some points relevant here, though 
there are substantial differences between educational and employ- 
ment selection. I shall, however, make one point. 

It is important to note here that the Federal Executive A^iency 
Guidelines on employment selection issued in 1976 refer It) the joint 
test standards and, in fact, defer to-these standards for technical de- 
tail. Thus, executive agency guidelines effectively having the force 
of law are adopting professional standards that were not specifically 
written with this purpose in mind. If. in fact, the new uniform 
guidelines rely heavily on the joint test standards, and if the uniCorm 
guidelines are construed «Hi having the force of law . then a complete 
review of the testing standards will be in order to determine that 
they are suitable for this intended purpose. It is entirely appropriate, 
and indeed highly desirable, that federal regulations defer to 
professional. standards. Wlien this happens, however, we must be 
sure that the standards c^in bear this increased responsibility. 

The Law and Testing Irregularities 

Now ^ turn to the question of the reporting of i regularities in test 
adjpinistration. This . general term covers a r.ultitude of possible 
kinds of specific events, but the one of primary concern is cheating. 
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In some situations, proof of cheating is fairly solid. An examiner 
may one examinee copying from another or using ufiauthonzed 
.mat.eriab and may have the observation confirmed by a second 
examiner, or handwriting analysis or comparison of fingerprints may 
establish lhal.an examination has been taken by an impersonator. 
But in most cases, an allegation of cheating must be confirmed on 
a probabilistic basis, often through the statistical comparison of 
responses of two or more examinees. Once such evidence has been 
evaluated, a mechanism must be available ta provide the examinee 
wiifTappropriate due process in the review of this evaluation and 
then,Mf the allegation is confirmed, an appropriate process must be 
used to notify the concerned institutions of these findings. 

Two of the major testing organizations have well-developed 
policies for the handling of incidents of testing irregularity. These 
policies are designed to protect the interests of all parties involved. 
And, indeed, it is in the best interests of all parties, especially 
examinees, that testing procedures are not compromised. However, 
policies on irregularities are always written by the testing organiza- 
tion and or the institution contracting for the testing program. One 
mterested party, the examinee, is not directly represented; his only 
resource is through the courts. Not surprisingly, there has been 
some litigation in this area in recent years, though I know of no case 
m which yn examinee has completely had his way in court and 
prevented a testing organization from withholding his score or 
cancelling a reported score. However, in K.D. v. in\S (6). the New 
York State Supreme Court (New York's lowest court of general ju- 
nsdiction) did insist that in cancelling K.D.'s score, i rs would be re- 
quired to do so without giving reason for this cancellation. One rul- 
ing in one case m one lower-level state court does not establish very 
much precedent. But ii is clear that, the content of and manner in 
which irregularity information is transmitted from a testing organiza- 
tion to an educational institution will be the subject of close legal 
scrutiny. 

In the relationship between the educational institution and the 
student, the concept of /// /<>(<> parctuis is dead but it is not at all 
clear what has leplaced it. Some have argued a trusteeship theory; 
others have argued for a direct or implied contract relationship. It 
will probably take time to settle this issue. Moreover, in the testing 
situation, vve have a third party, the testing organization, that 
contracts with the educational institution to provide a service but 
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collects the money from the examinees taking its tests. Further- 
more, with respect to the contract between the testing organization 
and the representative of the educational institutions, the negotia- 
tion is essentially between equally strong parties. The individual 
student, however, is powerless. He must either accept whatever 
terms are established or forego the educational opportunity for 
which the test serves as gatekeeper. An examinee taking one test 
administered by Educational Testing Service must sign a statement 
to the effect that he recognizes that ars 

[reserves] the right to cariLcl or withhold any test scores if. in our sole 
opinion, there is adequate reason to question their validity, or if there 
are grounds for believing that a person has engaged in the act of 
dishonesty with respect to the testing process [Hmphasis added ) 

The question, then, is what recourse does a student have if, 
hypothetically of course, the testing organization exercises this 
assumed authority in uhat the student considers to be an arbitrary, 
capricious, defamatory, libelous, malicious, punitive, or un- 
professional way? Well, that is a long complex story, which my 
colleague. William Buss of the College of Law of the University of 
Iowa, and I shall tell soon but not today. Lest I create a false im- 
pression, let me affirm that there is far more due process in the han- 
dling of irregularity cases in ihe program to which I have alluded 
than the adversarial natuie of the quoted statement would indicate 
An Lrs example is used here to make a general point because its pro- 
grams are so highly visible and because ns management is, as al- 
ways, so cooperative in providing information. 

What I should like to suggest today is that the writing of ^ 
professional measurement standards for the handling of testing ir- 
regularities and the codifying of examinees' rights of due process 
would be a veiy useful undertaking. Would it not be desirable for all 
testing programs in which cheating was a matter of concern to have 
in their announcements the statement that in the event of an allega- 
tion of irregularity . examinees would be afforded due process under 
procedure^ consistent with codified professional standards? Perhaps 
everyone here will agree with this proposed codification of examinee 
rights. If not. I suggest that we all consider carefully the lUterna- 
tives: , 

First, theie is the possibility of continuing litigation. While I do 
not feel that we should solve oui measurement problems in the 
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courts and refuse to predict the outcome of such litigation. I will 
predict that it will continue. In the areas of due process and 
consuhier rights, the public is restless and much less willing to ac- 
cept even the appearance of arbitrary administrative treatment. De 
Funis, Bakke, and K.D. challenged the system on important issues, 
and there will surely be others. You may already have heard the 
names DiLeo and Clancy. While the next legal challenge to the han- 
dling of an irregularity incident is unlikely to receive the attention in 
the press that the reverse discrimination cases such as De Funis and 
Bakke have commanded. I think it will most likely attract more at- 
tention than did K.D. 

Second, if the measurement profession does not adopt standards, 
there is some chance that Congress may do the job for us. In prin- 
ciple, I am not opposed to federal regulation of industries. But we all 
know the disadvantgages of federal regulation, and I therefore hope 
that it would be the response of those involved with educational test- 
ing to try to make suci regulation unnecessary . Congressman Har- 
rington's bill (HR 6776) to regulate the testing industry is well-moti- 
vated. I personally happen to think that a better job can be done by 
measurement specialists. Furthermore, if this job is done within 
the profession, we can keep our standards under continuing 
professional review . modify ing or adding to them when necessary. If 
this job is done by Congress or by delegation from Congress to a 
federal agency, the measurement profe>sion may have as little 
influence on these standards as they hav ; until recently had on em- 
ployment selection guidelines. If we must have fedeial legislation, 
we ought to have the kind that defei s to piufession-t! standards on all 
points of substance. 



Protecting All Participants in the Testing Process 

There are many issues relevant to the four topics I have discussed 
that will require codification in piofcssiotial standaids. but in my 
thinking, one predominates There aic three parties involved in the 
testing process— the examinee, the educational institution, and the 
testing organization. It has. foi example, been alleged that in- 
telligence testing has been used to tiawk minority chikiien in such a 
way as to limit theii intellectual development. If this is true, some 
action should be taken. But who is responsible * Is it the school 
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which claims that thes^ procedures are justified by the scientific evi- 
dence in support of the validity of the test? Or is it the test publisher 
who acknowledges the responsibility to produce good tests but 
professesjio lack responsibility for spegific uses of the test^^ and the 
specific decisions which are based on test scores? To what extent 
wan ^nd should test publishers be required to monitor the use of their 
tests or the scores they report? Should they be able to say simply 
that they only publish or give tests and that they have no responsi- 
bility for the decisions that are made? Alternatively, can testing or- 
ganizations institute mandatory licensing procedures that permit 
them to control the use of their tests, or is this restraint of trade 
and/or infringement on individual pn^ft^ssional practice? 

I suggest that these questions be answered only by the creation of 
legally and scientifically valid professional measurement standards 
that respect and codify the rights of all participants in the testing 
process. It is my judgment that educational testing has been a m<\jor 
force in the democratization of the educational proces'S. The 
codification of new standards in the areas 1 have mentioned will, I 
believe*, further strengthen this force. 
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Discussion 

A 

WiNTON H. Manning ' ■ ^, 

Senior Vice President for Development and Research 

Educational Testing Service 

When I was a youngster growing up in St. Louis on the banks of the 
Mississippi— not so terribly far from Mel Novick's Iowa domain— a 
favorite pastime was scaling stones. A boy or girl who could get 
t*iree skips was a star; if you could get four hops before the final 
splash, you were a superstar. Mel Novick would probably have 
earned the equivalent of an Olympic medal because he has taken 
careful aim and with characteristic vigor and style, scaled five cur- 
rent and crucial areas in which testing and the law intersect. 
Hfs Jpve skips are: 

1. funding allocations under Title I 

2. admissions policy and the Bakke case , 

3. guidelines for use of tests in employment selection and licensing 

4. fairness and due process in the treatment of students suspected of 
cheating" or similar irregularities r 

5. the general question of the allocation of responsibility (and ac- 
countability) in educational decision making, which is implicit in 
all the foregoing. 

(The last is the final splash, and authorities differ on whether that 
counts or not in the Olympic point count for the stone-scaling 
event.) 

The problem I now confront is how to discuss adequately 
Novick's tour de force in 10 minutes or so. Upon reflection, I de- 
cided the only answer I could give, if asked whether I could do it, 
is the one that a f(jllow St. Louis lad. Yogi Berra. gave when asked if 
the Dodgers would sweep the Worid Series. Yogi replied, "I can 
answerthat question in two words— IM POSSIBLE!" 

I have now used up two minutes establishing that I am a human 
being with warm memories of childhood idylls and that you, as a 
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sensitive and intelligent audience, should sympathize with me in my 
predicament. Nowadays these are not mere ritualistic rites of plat- 
form palaver because, as I think Roger Lennoo would agi'&e. any 
officer oT a large testing organization should probably assume that a 
substantial number of people in any audience he faces will believe 
automatically that he is not human, never had a c hildhood, and de- 
serves all the hostility he gets. 
I shall now scale my stone: 



1. Title I 

Mel has pointed outcome pitfalls in implementing an achievement 
test-based formula for allocating Title 1 funds among the states and 
school districts. The income-based formula for Title 1 allocations is 
an exceedingly complex formula. For example, it begins with the 
poverty-line basis for determining eligibility for federal programs 
which, in turn, rests upon the 1970 census data on state-by-state in- 
come level, size of family, and sex of head of family, and data on the 
number of 5-to- 1 7-year-olds in a family. On this foundation, special 
adjustments are introduced to take into account: 

L A percentage of the afdc children (that is. /Affluent Families with 
Dependent Children, the term applied to children of non-poverty 
or "wealthier" ftimilies); 

2. Some percentage of the childicn not in families but in non-state- 
operated placements, such as foster homes, private institutions, 
and so forth; 

3. Some percentage of the handicapped and migrant children in the 
state; 

4. Annual updating to reflect changes in ti e Consumer Price Index; 

5. And finally, adjustments reflectmg the individual state's average 
per-pupil expenditure for recent years (specifically what it was 
three years ago), the national average per-pupil expenditure, and 
past funding levels within the state. All of these aie also subject 
to change and demand frequent updating. 

Mel has rightfully pointed out that a move to an achievement test- 
based eligibility formuia raises a number of knotty measurement 
problems, but 1 am not sure that the present income-based model is. 
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on balance, any more satisfactory. In summary, I am personally niot 
as pessimistic as Mel is concerning the feasibility of allocating funds 
by tests. The best service I can offer to you who are interested in 
this question is to commend for your reading an excellent nie study 
just completed which is entitled **Using Achievement Test Scores to 
Allocate Titl^ I Funds/' 

The decision regarding which basis to use— income or ability- 
should rest upon value considerations rather than technicalities of 
measurement. Title I was originally conceived to benefit poor people 
by breaking the cycle of poverty through educational intervention. 
Congress has drifted to a confused middle ground in more recent 
years. If they wish to direct funds to educational need per se, then 
achievement test-based allocation systems make a great deal of 
sense to me. If poverty is the central concern, income-based eligi- 
bility seems relevant. It is a matter of social policy on which 
Congress badly needs to clarify its intent. 



2, Federal Guidelines for Employment Testing 

I cannot really find anything with which to disagree in Md*s dis- 
cussion of this matter, and I know that Wayne Holtzmap will discuss 
this at greater length this afternoon. So I will pass over thia quickly. 
Furthermore, I have recently presented a lengthy paper on just this 
topic, and once started on these issues, it would be hard for me to 
stop. In my aera paper (entitled "Educational Research, Test 
Validity and Court Decisions"). I said that a decade of experience 
with the APA Standards and the lloc Guidelines suggests that there 
is a serious need for »*eco'*ceptualizing the theory of test validation 
implied in these two documents. In fact. I offered the same judgment 
as that visited upon the, Emperor Galba by Tacitus: 

^'Omnium consensus capa.x imperii nisi iniperasset/\ which 
means in plain English: 

"Had he never been placed in authority, nobody would ever have 
doubted his capacity for it. * 

I hope I may be pardqned this pretentious allusion to the classic 
Latin, but ;^hen one is, suiVounded by lawyers, it is well to remind 
them that even psychometricians can use dead languages to make 
their point! 

/ 
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3. Faimett and Due Process in the Handling of 
Suspected Cheating Cases 



There are few more uncomfortable, unsatisfying, and complicated 
topic&^than this one. Mel was right to raise it, but I don't think he did 
it justice, as J.hope he would agree. I look forward to reading the 
paper he has promised us on this matter. I would simply add some 
points that he overlooked, in order to expand the context of the dis- 
cussion: 

• At ETS, no student tested in the United States ever has his scores 
cancelled, regardless of how extensive the evidence of alleged 
cheating, without first being offered the opportunity to take the 
test again. If he or she does agree to be retested and the prior test 
performance is confirmed, the matter is closed. It is also relevant 
to add that the Appellate Division Court in New York (a higher 
court than that which Mel mentioned in the /w.D. case) has com- 
mented favorably on the practice of offering a retest, stating that 
spending two and a half hours taking the test again seems not to be 
an undue burden on the student. (Incidentally, it is probably a lot 
more generous treatment than is typically given to students by 
many professors, although I am not accurately informed about 
practices at the University of Iowa in like circumstances.) 

• ETS has set forth in one document a clear statement of the prin- 
ciples and procedures we follow in cases of irregularity— the due- 
process considerations, if you will— and this document is pro- 
vided to every candidate who finds himself in a situation where his 
scores are questioned. 

/ 

• In the Law School Admission Test program, a procedure involv- 
ing arbitration is employed, thus reducing time and expense for 
the candidate that a court case might require. The law school 
program is. by the way, the only et's program in which a reason is ! 
given for score cancellation, a policy adopted by the Law School 
Admission Council after careful deliberation. I realize this does 
not address all the issues Mel has raised, but it is important to 
dispel, to a degree, the impression that his understandably brief 
comments may have left with many of you. 
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4. Fairness in Admissions, as Exemplified in the Bal(l(e Case 

This is by far the most important and complex issue that Mel has ad- 
dressed. As one of the authors of the recent Carnegie Council report 
on the subject of race in adnfissions to higher education, it is 
necessary that I first set forth clearly where I come from on this mat- 
ter before turning to the more provocative elements of MePs dis- 
cussion, ; . ^ ^ 

The central social and educational issue of the Bakke case is the 
problem of balancing considerations of individual and group equity, 
* a problem which turns upon difficult value choices. Not all indi- 
viduals and not ail institutions will agree about them. In this circum- 
stance, the public must have confidence in the process by which de- 
cisions are made. The selective professional schools in particular 
must be prepared to face public scrutiny of their processes and their 
policies; and both the processes arid the policies should conform to 
their own missions and to the demands of public policy and should 
be fair, as among individuals similarly situated. Above all, these 
schools must be concerned with making optimal use of their 
facilities to develop human resources for service to society. In the 
effort to rea:h this»goal. racial e.^eperience is relevant within the ad- 
missions process because important educational and professional 
objectives will not be attainable unless, as colleges and universities 
go about the task of making admissions decisions, consideration is 
^ i ven to the m in ority s ta t us of in d i vidual dpplic 'a nts. 

Mel has suggested that the Supreme Court's decision may require 
that w'ays other than the use of ethnic or racial quotas must be 
devefoped to help eliminate the wide disparities in educational ad- 
vantage of young Americans. He then goes on to say, and I quote. 
''The only way I can see this happening is through the quantification 
and scaling of a measure of individual disadvantage, and the alloca- 
tion 9f funding and individual attention being based on this 
measure/* (Emphasis supplied.)' 

I have no quarreTwith the principle of allocating funds on the basis 
of an aggregated measure of disadvantaged status, or even its 
consideration within a context of examining a particular candidate's 



^Editor's note:. The sentence quoted above is from the draft of Mr Novick's 
paper that Mr. Manning used as a basis for his discussion The sentence was 
revised In the final version that appears in these Proceedings 
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background. I question, however, the desirability of making indi- 
vidual admissions decisions about particular students on the basis of 
such a standardized test of disadvantaged status. 

It is an intriguing idea, one that I have played around with a bit 
myself. In fact, I constructed such an index of disadvantaged status, 
based upon family income, parental education, and whether English 
is spoken in the home or not. and then applied it to data on 16,000 
students (4,000 blacks, 4,000 whites, 4,tJ0 orientals, and 4,000 His- 
panic students). There is a modest increase in the proportions of 
minority students found in the upper 10 percent of the distribution of 
test scores, after these factors are regressed out, but not much. I 
don't want to belabor this point; you can read about it in my Car- 
negie Council report (4). Frank Evans, using a similar logic^got the 
same disappointing results using law school admissions data (3). If 
you use socially disadvantaged status rather than racial experience, 
the proportions of minorities selected will be substantially reduced 
over present levels; disadvantaged status is no substitute for taking 
race or ethnicity directly into account if you are concerned about 
maintaining enrollment levels of minority students. 

But his is not the only point we should be debating, as, I am sure 
Mel would agree. The point is whether giving each child a score on a 
scale of disadvantagement is good social policy. Lc^-, it lead to the 
sort of society we want? Would you want to establish a quota, for 
example, for the number of students scoring below 500 on the 
Educational Disadvantagement Scale? 

In the effort to strike a balance between the rights of groups and 
the i;ights of individuals— betv^een "a pluialist society and a nation 
of individuals '— I fear thit the proposed instrument is too frail a 
bridge to bear the traflk. As an instrument for research analysis, as 
a means of dissecting the social consequences of aggregations of in- 
divijjual admissions decisions. I can find no quarrel with the Novick 
proposal. But I uould part company with Mel if uch a scale weie to 
become the explicit basis for admitting" and rejecting students. A 
writer in The New RepuhlU (2) recently put it well: 

To be classified is to he jmlgeil as a menihei. not as a parlicnlai 
person. Thai ?s doubly ihrealcPMig first As to imliMihial lights ant! then 
to the integrity of communit\ hfc Classification foi a purpose other 
than sheer lescnpuon |the analyt^^ function to uhich 1 have aMudedl 
implies a i.. catchy of classes. So it diives people lo choose their 
groups, if the> can. for leasons othif than their private feelings and 
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commitments . . . Admissions pfficcrs should look for personal 
strength— pride, energy, enierpriic, compassion— with the under- 
standing that these qualities arc differently expressed in different cul- 
^ tures, anji^ tested far more harshly in some parts of our society than in 
others. But personal strength, by definition, is an individual trait, not a 
group trait. It cannot be recognized unk*ss all groups are treated as indi- 
viduals. It is difficult to do that with any degree of fairness in an egali- 
tarian society. But thai is what doinfijitstid' ra/itircs. [Italics supplied.] 

Simple justice, I believe, requires that admission Officers take 
racial experience — particularly evidence of having surmounted bar- 
riers of racial discrimination — into account as they look in depth at 
each applicant. The explicit use of scales of disadvantaged status as 
a part of a student's credentials should \k approached with great 
caution. 

We Must Proceed Slowly 

Twenty-two years ago. Professor Edmund Cahn, writing in they4;i- 
fmal Survey of American La\\\ observed that "we should not make 
our constitutional liberties a function of anybody's science." Law 
remains free because the principle upon which it rests is an ideal; 
measurement, broadly conceived, is no more than the systematic 
quest for regularities in human behavior ( I). Although it is inevitable 
and necessary that law and social science consort with one another, 
their interaction is fraught with peril to each. 

I wish we had a whole day to talk about Mel Novick's provocative 
paper. We might even skip a few stones as we follow his own injunc- 
tion — namely . that "those of us dedicated to the advancement of the 
testing profession will need to proceed slowly to try to make some 
coherent sense out of the legal confusion ..." 
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President, Hof^f^ Foundation for Mental Health and 
Hog^ Professor of Psychology and Education 
The University of Texas at Austin 

For several generations, psychologists and other specialists in 
educational measurement have struggled with the concept of test 
validity and hov\ best to determine it. The basic idea is deceptively 
srmple. When we raise a question about a test's validity, we want to 
know whether the test really measures what it purports to measure. 
How valid is the test for a particular decision or interpretation 
proposed for it? No matter how excellent a test is in other respects, 
if It measures the wrong thing or is wrongly interpreted, the test is 
worthless. Our chief concern here is with the use of tests and educa- 
tional measures for making decisions that affect individuals. The 
kinds of questions ordinarily asked are the following: How valid is 
tnis test for deciding whether or not an individual is qualified to enter 
law school? Can I use this test for making a decision as to whether or 
not an cight-ycai-old child should be placed in a special education 
class for slow learners? Is this test a good one for selecting the best 
applicant for a limited luimbei of job openings in my company? The 
more valid a test is for a particular selection or classification 
pioccdurc, the moic likelv one can avoid making mistakes in using it 
as a basis for decisions about individuals. 

The lapid giowth of testing as a means of personnel selection and 
classification during World Wai li clearly demonstrated that even 
tcst> with iclatively low validity can be of great practical use if one is 
able to select only a small number of individuals from a large number 
of applicants. If one selects only 50 individuals fc^r flight training out 
of 5(K) who wish to become pilots, the important consideration is to 
reject all the doubtful cases, taking only the' ci earn of the crop. 
When the cost of tiaining an An Force pilot runs upwards of S 100,- 
000 pc! pilot, thcic IS obvious social value in using very high selec- 
tion ratios with extensive batteiies of tests to insure that the ones 
admitted to the tiaining piogiam aie the individuals most likely to 
succeed This classical model of selection and classification works 




Validity and tagatlty 



fairly well as long as one is not concerned about vhat happens to the 
very large numbers who are rejected in the process. The fact that 
many rejected applicants might indeed have been outstanding suc- 
cesses if they had been selected is irrelevant as long as those who 
are accepted do succeed. " ' 

During the 1950s and early 1960s, growing awareness of social in- 
justice in the United States led to sweeping new legislation such as 
the 1964 Civil Rights Act. Title Vll of this act was designed to 
achieve equality of employment opportunities and to eliminate dis* 
crimination in hiring and promotion practices. The act was a strong 
one leading to numerous challenges of employment practices, many 
of which involved the use of tests for selection and classification of 
personnel. Discriminatory purpose on the part of an employerneed 
not be proved, and it is insufficient merely to demonstrate a rational 
basis for the challenged practice. Shortly thereafter, the Equal Em- 
ployment Opportunity Commission developed its Guulelines on Em- 
ploymeni Selection. The Guidelines leaned heavily upon the 1966 
edition of the American Psychological Association's Standards for 
Educational and Psychological Tests and Manuals, a set of 
professional test standards developed jointly by the major 
professional organizations concerned with educational and 
psychological te«>ting. 

The EEOC Guidelines placed a heavy burden upon the test user to 
prove that there was sufficient predictive validity in the test, wheti 
used for hiring or promoting minorities as well as whites, to meet the 
requirements of nondiscriminatory selection. Other approaches to 
validation were generally set aside as insufficient. This strong pos- 
ture soon led to a series of law suits that have recently culminated in 
several important Supreme Court rulings. Not only Title VII of the 
1964 Civil Rights Act but also the Equal Protection Clause of the 
Fourteenth Amendment to the Constitution has played an important 
role in these Court decisions. Before examining these decisions in 
more detail as they relate to questions of validity, let's take a closer 
look at the ways in which test validity is generally established. 



Establishing Validity 

Determining the vi.lidity of a testing piocedure can be done in 
several diffeient vays. If the procedu/c is aimed at selecting indi- 
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viduals who will be successful on the job or in school at some future 
date, then the ideal method is to study a large number of individuals 
who have been given the test prioi to admission to see who the suc- 
cesses and failures are at a reasonable later point in time. Correla- 
tion between the test and the criterion yields a coefficient of predk- 
tive validirx. A second approach often considered a variation of pre- 
dictive validity involves obtaining both test scores and criterion 
measures on the same sample at the same time. If a new test cor- 
relates highly with an established valid test when given to the same 
subjects at the same time, torn urnnt \ lUdity is established for the 
new test using the previously validated test as a criterion measure. A 
third approach, known as unuvnt vaiidit}. is particularly applicable 
when no criterion measure is available and ihe focus of concern is 
upon measuring certain knowledge or ab4[t> . the relevance of which 
is self-evident. A fourth method of cstabTi^hmg validity involves 
relating the test scores to a theoretical concept or construct' by 
conducting a series of interlocking cxpenments designed to test 
hypotheses based upon theory. Such lonstnat validity is only 
slowly established as hypotheses aie confirmed, refuted, or revised. 
All four of these methods aie well established as fully acceptable 
procedures, but one must he careful to fit the method to the 
particular purpose he has in mind. 



Establishing Predictive Validity 

For selection procedures in which indniduals are admitted or re- 
jected on the basis of a test scoie, establishing piedictivc validity is 
of primary concern. There aic foui conditions that must be met in 
order to establish predictive validity foi a decision rule based upon 
test scores. First, one must have a good criterion against which 
performance on the test can he compared. If the purpose of the test 
to select individuals who will succeed in graduating from college, 
one must wait at Ictist four years fioni the point of .idmission until 
the criterion can be ohtarned. If the test is to be used for selecting 
the best applicants for a job. t)htainmg adequate criterion measures 
may be even more difficult, depending upon what is most highly 
valued in the way of job performance. In most cases. ther*e is no 
single criterion measure that is satisfactory . Criterion measures may 
be just as multr-dimerisiona! as the tests used to predict them. A test 
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may prove to be highly valid for predicting one criterion and quite 
unsatisfactory for predicing another within the same selection 
procedure. If one insists that a test must predict later success in a 
career rather than merely success in completing a training program, 
a delay of many years may have to occur before both test perfor- 
mance and criterion performance are measured in order to validate 
the test. 

A second condition of importance in establishing predictive 
validity is the admission of a full range of test scorers, both high and 
low, for whom criterion data are obtained at a latei date. When a test 
is used for selection so that only the top applicants are admitted, 
there is no way of determining later whether those who were re- 
jected would htjve been failures. In most practical situations, it is 
very difficult to maintain sufficient heterogeneitv for an adequate de- 
termination of predictive validity. 

Where serious questions of bias or unfairness arise in a testing 
procedure, as in the case of a test which may be judged biased 
} against blacks in favor of whites, one must have a sufficient number 
j and heterogeneity of the disadvantaged individuals as well as those 
who are favored in order to establish separate validity coefficients 
for both classes of individuals. This third condition, vigorously 
pursued by the Equal Employment Opportunities CommissiprOis 
particularly difficult to meet in most practical situations. In (indus- 
tries where blacks have been discriminated against until recently, 
there is an insufficient amount of evidence from previous research to 
defend the test procedure against charges of bias. 

A fourth condition for establishing predictive validity in a selec- 
Uon pioceduie follows from the requirement tli^at a good criterion 
measure must be obtained at a latei date against which the test 
scores can be compaied. The lapse of time between test scores and 
cnterion data must be sufficient to allow for the development of an 
adequate criterion measure. During this peiiod, which may range 
from months to yeais, there may he con^Jerable attrition due to 
dropouts or turnovei in personnel, if this attrition rate is appreciably 
different foi whites than it is for blacks or Chicanos, the outconie of 
the validity study is considerably weakened. 

For all of the above leasons, establishing predictive validity is ex- 
4:eedingly difficult and, indeed, impossible in many cases whc^e tests 
are used. The best studies aie usually those carried out under, the ^ 
auspices of majoi social institutions with thousands of individuals 
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participating, such as in military or educational settings. But more 
often than not, the sample sizes are too small to determine the 
practical significance of the "selection procedure; there are too few 
cross-validation studies to determine how strong the validity is 
under varying conditions: there are too few disadvantaged indi- 
viduals in the validation study because of a previous history of dis- 
criminatory employment practices: and weak or irrelevant criteria 
are employed. Given this state of affairs, it is small wonder that 
vigorous enforcement of Title Vllinthc 1964 Civil Rights Act. as in- 
terpreted by the Equal Employment Opportunities Commission, has 
resulted in a number of federal court cases involving the use of tests 
in personnel selection and classification. Three of these employment 
test cases have been carried to the Supreme Court: Gri^^gs r. Duke 
Power Company^ in 1971 . Alhenuirle Paper Company v. Moody- in 
1975. and Washington v. Davis^ in 1976. It is instructive to examine 
briefly each of these cases with respect to their implications for test 
validity, and legality. 



Court Rulings on Tests ' 

In Gri^'gs, black lalSorers at the Duke Power Company's generating 
facility filed suit against the company . challenging the company's re- 
quirement that an employee must have a high scht)ol diploma or 
must pass an intelligence test in order to be transferred from the 
Labor Department into one of the othei four departments at the 
power plant. Since blacks were employed onlv the Labor Depart- 
ment where the highest pavingjobs had lower Aagcs than the lowest 
P«^yiP8 jobs in the other foui departments, it is clear why Title VII of 
the Civil Rights Act was used as a basis for this lawsuit. The 
Sjpreme Court ruled in favor of the Negro employees. The Civil 
Rights Act requires that a selection test which discriniin*ites on the 
basis of race is prohibited unless the test can be shown to be clearly 
related to job perfoimance. While the buulen of proof for showing 
discriminatory consequences rests upon the plaintiff, once such 
proof iias been picsented. the burden shifts to the defendant who 

'401 U S 424(1971) 
■422 U S 405(1975) 
M26U S 229(1976) 
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must prove that the test does indeed Correlate with job performance. 
As Lerner (1) has pointed out in a review of these employment test- 
ing cacss, the lower courts quickly applied the Cn'm definitions to 
cases arising out of very different contexts. The net result was 
considerable distress and confusion ovePHhe differing bases for 
proving discriminatory effects on the one hand and for proving test 
validity with respect to job performance on. the other. While the 
\ Supreme Court ruling four years later on the Albemarle case pro- 
/vided some clarification, it wasn't until Wasliinston v. Davis that 
significant new features were considered, leading to at least a partial 
resolution of the problems raised by these lower court cases follow- 
ing Gnm- ' , 
The action in W,ashingU)n v. Davis started in 1970 when two black 
police officers filed suit against the Commissioner in the District of 
Columbia alleging that the promotion policies of the police depart- " 
ment wereVacially discriminatory. The department's recruiting' 
procedures involved a written personnel test that excluded a dispro- ^ 
portionately high number of black applicants. The test had been 
developed by the Civil Service Commission and was designed to 
measure verbal ability, vocabulary, reading, and comprehension. 
The district court upheld the use of the test by the police depart- 
ment, but the appeals court revcrsed the decision, ruling that the test 
had not been established as Job related/* In 1976. the Supreme 
Court reversed the appeals court, ruling in favor of the police de- 
partment and its use of the test for selection purposes. 

The §Upreme Court's written opinion in Washitii^ton \\ Davis is of 
particular interest because it includes detailed reference to test 
validity." The Court noted tKat there is no single method for appro- 
priately validating employment tests in^elation to job performance. 
A validation study of the test had been completed by the Civil 
Service Commission, which demonstrated a relationship between 
lest score and performance iathe recrjyit training program for police 
officers. The Supreoie Cmirt joined ^'the earlier district court in 
concluding that this evidence was sufficient under the Equal Protec- 
tion Clause of the Fourteenth Amendment. One month later, the 
Supreme Court extended these concepts further by denying plain- 
tiffs petition for review of Tyler v. Vickery, a case concerning the 
use and validation of the Georgia Bar Examination which the plain- 
tiffs claimed was racially biased and invalid. As Manning (2) pointed 
out in a recent analysis of'tcst validity and court decisions, the Fifth 
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Circuit Court of Appeaft^in Tyler v.^Vickery revealed a new disposi* 
tion of the courts regarding what constitutes evidence of test validity 
in cases in which a dispropoptionate number of blaCk applicants, as 
contrasted to whites, fail an examination. The petitioners who had 
been denied adrnission to- the bar by the test argued that the bar 
examiners could justify the use of their test only if they were able to 
show a predictive validity correlating test scores with later occupa- 
tional performance. Expert witnesses for the bar examiners differed 
sharply on the issue of test validity, claiming that the bar examina- 
tion would be sufficient if it had content validity. The court rejected 
the arguments-of tfie petitioners — namely, that the user of a test ^ 
with discriminatory results must prove the job-relatedness of the 
test — and held that the Equal Protection Clause of the Constitution 
requires nothing more than that an examination test skills and 
knowledge which have a logical, apparent relationship to those 
-necessary for the job. For the first time, conti*ary to the Guidelines 
of the Equal Employment Opportunities Commission, the courts re- 
jected exclusive reliance on a criterion-related approach to the proof 
of job-relatedness, ruling that the proof of content validity would be 
sufficient. 

An even clearer case supporting content validity is the United 
States v. South Carolina,'* as decided by a three-judge district court 
in 1977. The South Carolina State Board of Education had been us- 
ing the National Teacher Examinations to certify and determine the 
pay levels of teachers in South Carolina. The test had been used in 
South Carolina for over ."^0 years, and a considerable amount of test 
data had accumulated on both black and white te^acher performance. 
Extensive validation studies had been undertaken by Educational 
Testirig Service which resulted in different minimum test scores in 
various areas of teaching specialization. Th\; plaintiffs challenged 
each of these uses of the National Teacher Examinations, contend- 
ing that more blacks ;han whites historically had failed to achieve 
the required minimum score. In its opinion, the court cited in detail 
the design of the validation stud> and ruled that it was sufficient to 
meet the burden placed on defendants under Title Vll of the Civil 
Rights Act. The study demonstrated content validity by measuring 
the degree to which the content of the tests matched the content of 



^In the District Court of the United States for the District of South Carolina, 
Columbia Division, Civil Action No 76-1610. filed April 14. 1977 




69 



ValldKy.and Legality 

the teacher-training programs in South Carolina. A minimum score 
requirement in each of the m^or fields of specialization was es- 
tablished by estimating the amount of knowledge that a minimally 
qualified teacher candidate would have in South Carolina, A total of 
456 teacher-educators in South Carolina from both races par- 
/ ticipated in the content validation of the National Teacher Examina- 
tions. The arguments presented go /urther than any other court 
opinion in supporting content validation as sufficient, even in the 
f^e of unintended discrimination. How United States v. South 
Carolina will fare, when and if it is appealed remains be seen. 
Regardless of final' outcome, conflicts between the eeoc Guidelines 
for test validation and rulings by the federal courts have increased 
sharply in the past 18 months since Washington v. Davis. 

What do these trends of the past six years suggest with respect to 
^ test validity and legality? Clearly there is less burden now upon the 
test user to establish the predictive validity of the test procedure 
;han there was several years ago when the lower courts were ex- 
tending Gfiggs into unknown territory and when the eeoc 
Guidelines for test validation prevailed. For the reasons cited 
earl er, good predictive validation studies are difficult, if not im- 
possible, to achieve in many cases. The use of intermediate criteria 
in tr aining situations rather than job performance, the adoption of 
multi-dimensional criteria as well as multi-dimensional assessment 
proctiures where feasible, and a heavier emphasis upon content 
validation when dealing with specific knowledge or skills can 
provide a stronger basis for accepting or rejecting a testing 
procedure th^m the deceptively simple unidimensional prediction of 
a question'.tble criterion. As Messick (3) has pointed out in a dis- 
cussion of' meaning and values in measurement, advances will be 
made only by adopting a more comprehensive framework than 
either predictive, concurrent, or content validity can provide when 
considered alone. For Messick, all measurement should be linked to 
constructs which serve to organize disparate findings. Proof of 
validity must rest upon both empirical and logical analysis if it is to 
prove sufficiently jobust to withstand challenge in the courts. 

Conclusions 

In conclusion, what can be said at this time concerning the relation- 
ship between test validity anc! legality '^^ In my opinion, the following 
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points are worth noting: 

K A testing procedure that incidentally discriminates against one 
or more classes of individuals can only be justified if job-rclatedness 
is demonstrated. Once proof of discrimination has been 
demonstrated by a plaintiff, the full burden of proof of test validity 
rests with the test user as defendant. The test user does not have to 
prove the job-relatedness of the test by a predictive or concurrent 
validation; a logical, apparent relationship between the test skills or 
knowledge and the job by content validation is sufficient. 

2. An excessively heavy burden upon the test user to prove job- 
relatedness by predictive validation has been lightened 
substantially. The burden of proof is present, but a greater choice of 
alternative validation procedures has been recognized by the courts. 

3. Predictive validation of a testing procedure for personnel selec- 
tion is still the method of choice when a good criterion measure can 
be obtained, when adequate samples of appropriately heterogeneous 
individuals are available, and when differential attrition due to ex- 
traneous factors is not markedly different for identifiable classes of 
individuals such as whites, blacks, and Chlcanos. 

4. Immediate or intermediate criteria can be a satisfactory sub- 
stitute for ultimate job-related criteria in concurrent or predictive 
validation of a testing procedure, provided the relevance of the sub- 
stitute criteria can be demonstrated. 

5. Content validation is a fully acceptable procedure for person- 
nel classification using tests of knowledge or skill (a) where the sam- 
ples of items or tasks to be performed can be demonstrated as 
reliably representative of the specific domaui of the abilities in ques- 
tion, (b) where there is expert consensus that the domain is relevant 
to tho training program oi job requirements, and (c) where irrelevant 
difficulties have been eliminated. An example of an irrelevant 
difficulty is the inadvertent requirement of unduly high verbal ability 
to understand items in a written test of mechanical ability for selec- 
tion of auto mechanics. 

Undoubtedly, additional court decisions will be necessary to cla- 
rify further the complex and evolving relationships between the 
validity and legality of selection and classification procedures 
involving tests. Decisions about human beings are clearly too im- 
portant to be left primarily in the hands of psychologists or educa- 
tors, regih'dless of advances in psychometric theory and practice. It 
is incumbent upon all of us to listen carefully to what the courts have 
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to say. Recent decisions and their supporting opinions have once 
again broadened the range of recognized options for validating test 
procedures. Al the same time, the limitations of traditional tests 
with their heavy emphasis upon verbal skills have become increas- 
ingly apparent. Only by the accelerated development and validation 
of new techniques for measuring oth.er important personal charac- 
teristics can we hope to acfhieve both social justice for the disad- 
vantaged and equal protection for all individuals. 



1. Lerner, B. Washington v. Davis: Quanlily. qualiiy and cqualiiy in cm- 
ploymcnl icsling. In Philip Kurland (Ed.). .S///^r<"mt' Couri Ri'yU'^^' ( 1976). 
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Two considerations make any intelligent discussion of the social 
implications of recent court decisions most difficult and perhaps 
speculative: For one. the possible resolution of the conflict between 
equal access to educational opportunities and the protection of indi- 
vidual rights is still pending in the U.S. Supreme Court (Bakke v. the 
Regents of the University of California), Future consequences, to a 
great extent, will depend upon how the Court decides and whether 
the decisions should be given a narrow or broad interpretation. 
Second, one must realize that any efforts to identify all or even a 
substantial portion of the malicious or beneficial consequences of 
perhaps the most critical decision before the court since Brow n will 
surely fall short of Its mark. 
^ Thus, armed with these caveats. 1 now will attempt to venture into 
these murky waters, cognizant that the rocks underneath on which I 
must tread make for slipper> footing. Houever. I have taken the 
precaution of interjecting the words "some possible" in front ofthe 
title of this presentation. 

One strategy for determining the possible consequences of recent 
litigation involving access to opportunities for minorities is to ex- 
tract the common elements reflected in the concerns ofthe more 
vocal opponents and proponents of affirmative action programs in 
general and special collegiate admission programs in particular. 



^The author, who is currently on leave from Indiana University, assumes full 
responsibility for all views expressed herein none of which should be construed 
as reflecting the policies or beliefs of RTI or Indiana University 
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Another approach is to focus, by logical analysis, on the specific im- 
pact that an unfavorable decision by the Supreme Court may have 
on various elements of affirmative action. Admittedly, the latter ap- 
proach rests on the assumption that the idea of affirmative action in 
principle is neither repugnant nor unconstitutional. 

Before addressing these concerns, it may be best I) to turn briefly 
to the "compelling reasons" for the past legislative responses 
designed to increase the likelihood of opportunities for minorities 
(the ''compelling reason" principle, as we shall see, has played a 
crucial role in receift court decisions); 2) to outline the mprc salient 
federal laws instituted to meet these needs; and 3) to outline the 
trends in litigation that have emerged because of either voluntary at- 
tempts to meet the social need and the spirit of the law or because of 
policies and practices that have intentionally or unintentionally cir~ 
cumvented the law. 



The "Compelling Reasons" 

The devastating effects of slavery and past and present racial and 
sex discrimination have been documented and raiel> denied. Nor 
has the generation-to-generation link between poverty . despair, and 
educational and psychological disadvantages among" racial mi- 
norities been re&ited. 

To be sure, ethnic minorities such as blacks. American Indians.* 
and Hispanics, as weil as women, have made significant advances in 
recent decades. For example, statistics jon white collar employment 
show gains for nonwhites and Hispanic Americans during the 
decade of the 1960s, unemployment dropped from approximately 1 1 
percent in 1961 to less than 5 percent u t%9 for non whites 20 years 
of age and over: and the rate of participation by women in the labor 
force increased from 33.3 percent in 19.50 to 44.7 percent in 1973 
-(21). 

However, in spite of the gains, the overall employment picture for 
minorities and women remains discouraging. In terms of the 
percentage of the civilia.i labor force that is unemployed, little has 
changed: The nonwhite, white ratio of unemployment was 2: 1 in 1973 
as it was in 1954 (25)\ the median income of blacks stood at 58 
percent of the white median income in 1973. the same as found in 
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1954' (22); women's 1973 earnings were only 58-60 percent thai of 
men's income (23): and women and minorities condnue to be under- 
represented in the high-paying prestige jobs and overroprcsented in 
employment requiring fewer skills and, of course, paying the lowest 
wages (6); precipitated by the 1974 recession, the jobless rates 
reached 14.3 percent for nonwhites and 12.4 percent for Spanish 
Americans in 1975 and then peaked in 1976 at 14.5 percent for non- 
whites. 

However, the most disturbing and incendiary statistics are the 
jobless rates among minority teenagers of 16-19 years old. In 1954. 
the unemployment rates for nonwhite male and female adolescents 
were 14.4 and 20.6 percent, respectively; by 1964, the rates had 
climbed to 24.3 and 31.3, respectively; and by the first quarter of 
1975, black teenage unemployment had climbed to 39.8 percent 
compared with 18 percent for white teenagers. (Unemployment for 
white teenagers has been rising steadily, ranging from approxi- 
mately 12 percent in 1954 to 14.1 percent during the third quarter of 
1974 [17].) In the large urban centers, the estimated unemployment 
for minorities, particularly blacks, may be as high as 50-6(^percent. 

These statistics are alarming for several reasons: I) A large 
number of the 16-19-year-olds, though closer than their older 
counterparts to the usual point of enlr> to higher education x)r job 
training, are receiving neither. 2) the lack of training and empluy- 
ment at these young ages will likely as*^ure the cyclical pattern of 
poverty that typically characterizes the urban and rural minority en- 
vironment.^; 3) the feeling of hopelessness about obtaining employ- 
ment and the fiustration of attempting to live from day to day, 
which is now festering in the cential cities! provide all the necessary 
ingredients for a tragic rerun ofthe crisis of the 60s as well as provid 
ing the imp'etus for increased crime and its concomitant social costs. 
Data from the National Longitudinal Study (nls) of the High School 
Cla^s of 1972 sponsored by the National Center for Education 
Statistics (c i s). illuminates this point. The m s is a long-term educa- 
tional research piogi,im designed to liack a national probability 



2|n 1975 the median income for black families rose to 61 percent of the median 
income for whites However, the gam was due to a smaller income gam for 
whites between 1974 and 1975 than for black households Median income for 
whites increased by only 6 percent in contrast to 10 percent for blacks (20) 
Moreover, white female mcome traditionally has been lower than income for 
white and nonwhite males, with black females reflecting the lowest wages (24) 
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sample of 23,000 young men ahd women who graduated from high 
school in 1972. The inient is to examine their educational and occu- 
pational patterns, plans, aspirations, and attitudes and to relate such 
information to the samplers prior etiucational experiences and their 
personal and socioeconomic characteristics. A recent report (13). 
noted that unemployed whiles tended to say that they were either 
going to school or did not want to work; both blacks and those of 
Spanish backgrounds more often mentioned a shortage of jobs, 
inadequate training, or lack of experience as reasons for not work- 
ing. 

Enrollment statistics, like epiployment data, also provide a social 
indicator of the extent to which we are meeting the needs of our 
people. An examination of college enrollment data reveals that the 
enrollment of women and minorities has increased dramatically in, 
recent years; Women now constitute aBout half of the' first-year 
graduate enrollment in most institutions of higher education, accord- 
ing to 1975 US Census Bureau data (9). Between 1970-1975. the 
enrollment of women in graduate and professional schools rose 
about 75 percent, while the enrollment of their male coimterparK' 
increased only 23 percent. Primarily reflecting the impact on^fge 
enrollments, the number of women receiving doctorates inofx^ased 
59 percent— from 4,600 in I971to7,300in 1975 (male doctt^/ates de- 
clined 2.6 percent during this period, from 27.500 to 26.800); 
women's first professional degrees, in the same period, rose dra- 
matically by 184 percent to a total of almost 7.000. Yet. Statistics on 
declining enrollments after the first > ear suggest that women are less 
likely than men to receive their bachelor's and advanced degrees. 
Moreover, in terms of the number of doctorates cojiferred in 
selected fields, recent studies by the National Center for Educa- 
tional Statistics show that in 1975. women received only 21 percent 
of the awarded doctorates. 13 percent of the first professional 
degrees in medicine, and 15 percent of the first professional degrees 
in law (14). 

Enrollment data for minorities also sliowetka marked increase. An 
Office of Civil Rights survey showed that minority enrollment 
in colleges and universities rose 11.7 percent between M72-I974. 
while total enrollment in the Scime period increased by only 2 per- 
cent. These increases were across the hoard for all disad- 
vantaged minority groups— American Indian, black. Asian- 
American, and Hispanics. In 1972. the proportion of minority-to- 
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majority t'ull-lime total enrollment was 1 1 9 percent but by 1974 it 
stood at 13.1 percent. Freshman year statistics for 1974 showed that 
blacks constituted 10.8 percent of the total full-time enrollments; 
Spanish-surnamed, 3.5 percent, Asian- American, I.I percent; and 
American Indian, 0.7 percent. Moreover, from first-year under- 
graduate education through doctoral-level studies, black females 
consistently A:ompnsed a greater proportion of the total full-time 
enrollment than black males. More recent data suggest that minority 
enroMment has peaked and may be on the decline at predominantly 
black and white undergraduate campuses (19). Blacks have shown 
gam> m enrollment m the law and medical schools: a 111 percent 
inciease m enrollment in law schools in 1974 as compared with 1972 
and an increase of 50 percent in medical schools. 

In contrisr, the degrees conferred have not kept pace with 
mmonty enrollment. For example, in the 1973-74 school year, 
blacks were estimated to have received only 5 I percent of the 
bachelor's degrees awarded <8). Moreover, the percentages of 
cthrac mmtnitics joming the professional ranks such as lawyers and 
physicians remain small and unfortunately stable. For example, 2.2 
percent of all physicians were black in 1950 and in 1970(2). 

li one were to summarize the status of women and minorities in 
cmpk>vnient and college attendance, it could be safely concluded 
that inipressivc gains have been made in some aspects, but because 
of then low status ti) bcgm with, the g.'ins have not been sufficient to 
alk>vK for significant pu)gie><in representation in employment and 
higher cilucation for these uroups. And in some areas, particularly 
the iiib trainmg i>rthe i)ldei teenager i)i voiing adult, valuabL ground 
has been U)st 



The Response 

Ihe executive and legislative branches of the federal government 
have respiinded io the needs of individuals and groups that have 
been the targets of disciimination through various laws (lesigned to 
»tssuie their pri)leclK)n. Laws against employment discrimination 
wuc written inti) the C ivil Rights Act of 1866 and 1870 and the Hqual 
Pn»lecUon (JausC of the To'iiteenth Amendment. The Civil Rights 
Act o\ \^H>4 placed the federal g.)vernment in an affirmative-action 
posture bv pri)vuling the clenient of ^enfor cement so sorely missed in 
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Brown I and II (28); under Title IV of this act, the U.S. Com- 
missioner of Education 's authorized to assist ll v to desegregate 
and the Attorney Geneuri is- empowered to institute lausuils to bring 
about desegregation; Title VI forbids the use oFfedeial funds inan> 
federally financed program (such as apprenticeships, training, uork- 
study) that practices racial discrimination. Title VII, as amended by 
the Equal Employment Opportunity Act of 1972, prohibits discrimi- 
nation because of race, color, religion, sex. or national origin in any 
terms, conditions, or privilege of employ ment. including employ- 
ment in educational institutions. 

Other forms of antidiscrimination statutes include a) prohibitions 
against allocating differential pay on the basis of sex for thepeifor- 
mance of similar uork subject to the Fair Labor Standards Act 
(flsa) a,s well as protection for other personnel in executive, 
administrative, and professional positions not coNcied by i lsa (the 
Equal Pay Act of 1963), b) stipulations that require the establish- 
ment of affirmative action piogiams by all federal contractors and 
subcontractors with- contracts in excess of $50,000 and 50 or more 
employees, with the co. ractor subject to monitoring by an assigned 
federal compliance agency (Executive Ordei 1 1246 as amended by 
Executive Ordei 11375): and c) an extension of coverage of the 
Equal Pay Act that prohibits sex dlsc^min*ltion'again^t employees 
oi students of any educational institutK)n recc'ving fedeial financial 
aid (Title 9, Education Amendments Act of 1972). iMoreover, most 
stales and many local government agencies have established laws 
prohibiting employment discrimination 

Moreover, state and local governmental agencies, colleges and 
universities. «md some busmesses began developing programs 
designed to mci ease minority representation 

Resolving Conflict: Recent Court Decisions 

it was inevitaKie that the age old ci>ntlict between justice and 
equality wouKI finally reach the couils Only a bi lef desciiption of 
the Ihikki OI so-called leveiscd discnmmation case should be re- 
quired in light of Its vvulespieau publicity »uul gravity A yeai 
aflei its opening in 1968. the University ofC alifornia 'ledical School 
at Davis initiated the Lisk Wmlc Piogiam, a special admissions 
program that, between 1970 and 1974. admitted 71 minority 




Charles L. Thomas 

suidenls: 26 blacks. 33 Chicanos, and 12 Asian-Americans. Forty- 
nine additional minority students were admitted through the regiilai 
admissions process during these same five years— one black, one 
American Indian, six Chicanos, and the remainder Asian-Americans 
(4). Sixteen percent of the available places in the entering classes 
were reserved for the special admittees, applicants judged to be 
disadvantaged by the special admissions committee on the basis of a 
number of personal characteristics among vshich, as conceded by 
the university, included race. While certain elements of the ad- 
missions process were unclear,' several aspects weie apparent: At 
least for the 1973 and 1974 academic years, the undergraduate grade- 
point average (gpa) in science as well as the overall dPA for the spe- 
cial admittees were lower than for reg'ilar admittees, the average 
scores on the Medical College Admissions Test (mcai) were 
consistently lower for the special progiam participants,** consider- 
able overlap in college op\ (oveiall and science guides only) was 
evident hefween.the two groups of admittees, no white >tudent had 
ever been admitted (although it is uncleai whether any ever applied) 
to the program since its inception, and appaiently the applicants for 
the special program competed only amonj; themselves for entry 
rather than in a common applicant pool. What is also apparent but 
nut widely publicized is the fact that * n the basis of the hard data--> 



^Indeed, the amicus bnef filed by the Department of Justice pointed to several 
ambiguities (e g whether the final bench mark ratings of the special admittees 
were ever compared with those for the regular admittees and p'-'isely how 
race was employed in the admissions process) in the university s admissions 
p'ocedure in its argument for limiting the legal question before the court to the 
issue cf whether the University of California Medical School may consioer the 
race of applicants tor the purpose of operating a properly administered 
afftrmative action program (5) 

•^Scores for both groups of applicants were presented in percentiles, at least in 
the Department of Justices brief (3). but assuming roughly a normal distribu- 
tion and a standard deviation of approximately 100 on both subtests, it appears 
that the special admittees were roughly one standard deviation below the 
regular admittees on the verbal subtest for each year and for the years 1973 and 
1974, approximately 1 33 standard deviations and 1 25 standard deviations, 
respectively, on the MCAT science subtest These estimates of difference^ 
between the two group,> particularly for the verbal scores are consistent with 
other studies of standardired aptitute test score differences between minonties 
and nonminorities 
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that is the traditional measures of college gpa and mcai scotVs— 
Mr Bakke exceeded a hirae number i)f the reauhir (ulmillees as 
well. Therefore, the very question of status that prompted Mr. 
Bakke to seek redress in the courts appears to have been precipi- 
tated by the use of soft data such as the results of interviews and let- 
ters of recommendation. This final point may be moot with regard 
to the constitutional question of whether race can be used validly as 
one of the criteria for admissions. It is. however, terribly important 
to supporters of special programs who grou suspicious of the 
reasons why the program was singled out as the culprit when many 
factors in the rating process vveie equally vulnerable to questions 
concerning their relevance or validity in determining Mr. Bakke s 
final rating. 

It is not the intention here to argue the merits of ihe case but to 
emphasize the fact that the residual effects that have been generated 
in terms of suspicions, hostility, and so on ovei ihc clume <>J (Oses 
to settle this critical issue may match those predicted thi. morning 
if preferential programs prevail.^ However. . - ^s illustrate the 
extreme vulneiabilitv of such puig'ams the ki. .i ol careful sciu- 
tiny they may expect exen if the iiidgment of the Supieme Court it 
California is vacated. 

The case before the Supieme C ourt was loreoidained. I he setting 
was established at least six yeais ago. Ihe actors had only to be 
brought foiuard to centei stage. In 1974. a similar case before the 
Couit was rendeied moot because the individual in question finally 
was admitted and was neai giaduatum (12). However, even though 
this case was not deudcd. the prinupie o(cmp\os mg racially neutral 
admissions proceduies was voiced by Justice Douglas minority 
statement. Moreover, decisions similai lo those handed down by the 
California court in the Bakke case were handed down by the New 
York Court of Appeals last veai and the U. S. restrict Court 
Southern Distnct of New Yoik. this ve.ir in cases involving special 
minority-student piomams at medical colleges (1) Moie cuirently 
(and perhaps moie lumic). the (:o\oi.ii\o Supieme C^nirt la.t spring 
totfk under consideiatuM, a while studcnfs ch.im that the University 
of Coloiadii Law ScMuuil illegallv eu linl< d lum liom its speual ad- 

sindeed the university s appeal to the U S Supreme Court over the ob)ections 
of cm! rights leaders, labor unions, and public advocates that the case was not 
vvjii developed has raised some suspicion that a concerted effort was made \0 
apply the coup de grace to special admissions programs 
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missions progmni because of his race (11). In this instance, the 
question was raised whetKer an educationally andcultiTally "disad- 
vantaged'' white student has a constitutional right to be considered 
for admission under the special program designed to give preference 
to minorities, mainly blacks. Chicanos. and American hidians. Fi- 
nally, many of the current debates on the legal issues, social implica- 
tions, problems, and benefits of preferential collegiate admissions 
programs were embodied in the papers published b> the University 
of Toledo Law Review in 1970 (26). 



More Conflicts and Possible Implications 
for Affirmative Action 

The following two excerpts, the first fiom the dissenting opinion of 
Justice Harlan in' Fie ss\ v, /Vri/z/so/; and the other from the affirm- 
ing opinion of a federal district court in A s.sin iateJ Genend Contrac - 
tors of Massachusetts. Im . i. Altsliulcr {490 F. 2d9. 1973) concern- 
ing the "Boston Plan." an afnrniativc action plan uhich required 
contractors to employ a stated percci..age of minorities, point up the 
issues that have arisen in recent vears ovei the implementation of 
affirmative action programs: 

In respecl of liviI rights, LoniniDntt) all Liti/cns. the C i)nslitulion of the 
United Slates docs not, 1 think, permit .in> puhla aulhoitly to kmm the 
race of those entitled to be piotected m the eiijOMiienl i)f siKh rights 
- . Our Constitution is loIoi -blind and neithei kni)Us noi ti>li)rales 
classes among eili/ens 

It is by ni)v^ v^ell uiuIlm stood. ht»uevei. that oia society Liiniioi be tom- 
plelely Loloi-blind in the shoit teini if v^e aie to hti\e a Lolor-bhnd so- 
ciety in the long teini -\ltei Lenluiies of vie\Mng thiough Lolored 
lenses, eyes do not ^iifiLkh adjust uhen the lenses aie reiiioved. Dis- 
Lriniihcition has a Aa> of pLipetu<iting itsel!. albeit unintentional!) . bc- 
Liiuse the resulting inequalities niiikL nev^ uppoitunities less <»ssessible 
Preferential treatment is one paitial piesLiiption to leniedv our so- 
eicty's most intiansigent and deepK looted inecjUiilities 

Our nation, urestling with the cons^'tjtiences of its own past, has 
been and is'still attempting to plot a coui sc through the Scylla of the 
effects of racism and the Charybdis of individual injtistice; of at- 
tenipting to assure equality of opportunity and yet to pieserve our 
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democratic form oP meritocracy. In this struggle, there are many 
who express the view that the two opposing sources of conflict are 
more apparent than real; thai our democratic tenets and aspirations 
dictate nothing less than the elimination of race as a factor in bring- 
ing about equality of opportunity; that preference on the basis of 
race is always discriminatory, probably invidious, and perhaps un- 
constitutional; that, indeed, ^^justice and equality are more or less 
incompatible*' (7). Others noted that for 90 percent of the time that 
our great nation has existed, it has had institutionalized racism in 
some form or another and that the relatively short period since 
BroMv: cannot be vieued as sufficient for removing the effects of 
past racial and sex discrimination nor can it provide an equitable 
basis for minorities to compete with the majority now of in the near 
future. Race may be a proxy for other sins but ii can also stand on its 
own merits. Racial discrimination, it is reasoned, perpetuates itself 
and fequues more than laws and benign banalities. Others, view the 
nation as tiring of the Cough course and seeking refuge in rctrench: 
ment or, worse.^etrogression. 



Grounds for Agreement 

As the opponents and proponents state thoir arguments, there are 
some grounds of apparent agreement. One is that Bakkc coijjd have 
implications far beyond the matter of college admissions, perhaps 
striking at the heart of affirmative action Two: The decision, 
whatever it may be. will noi absolve the univeisity of its (►bligation 
to determine which couise it should take in fulfilling its mission in a 
democratic society . This is as it should be. for it is the prerogative of 
these institutions to establish their educational policies. However, if 
{be decision is in favor of Hakke. theie is the increasing possibility of 
additional lawsuits, 'jnce to many, discretion Js the better part of 
valoi. institutions of higher education may retreat entirely from spe- 
cial collegiate programs. Third: SiiTcere opponents and proponents 
agree that preference without i egard for ability oi personal qualifica- 
lions'is dysfunctional. However, there is lively debate over what is 
meant by ' qualifications'* or what the qualitative teaching role of 
the c '^ege should be. Opponents of affiimative action consider the 
affirmative action effort as that of a national discriminatory program 
that has had little effect on changing the status of racial minorities 
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and v<^omen because it rests on tenuous principles: Goals, no matter 
how they are explained by government bureaucrats, are viewed as 
quotas which, because of an undersupply of qualified individuals, 
can only be met by recruitmg unqualified persons. Moreover, such 
programs will reap scorn and additional feelings of inferiority upon 
the individuals for whom the programs were designed to assist and 
cause greater class strife as more and more interest groups come for- 
ward demanding their fair share (1.^). One of the most frequently oc- 
curring responses from academia is that the affirmation process is 
abysmally low in cost efficiency, is time-consuming and unproduc- 
tive in terms of paper work, and. worst of ail. infringes on the au- 
tonomy of the university and debases the quality of the institution. 
Therefore, it is^afe to assume that a decision'in favor of Bakke 
would be considered as going a long way lo redirjeet the country on a 
proper path toward true equality. Unfortunately, the programs ov 
policies that have been ofTered— and they have been only,a few — 
seem distressingly like afterthoughts and at best are totally un- 
realistic. 

Supporters of affirmative action, and theiefore for*the overturn of 
Bakke. rely heavjij cn the copipelling interest of the state in upgrad- 
ing skills and increa'iing the representativeness among niinoritiev 
and women in a ujde sps^clriim of educatit>lial and business institu- 
tions. Such consider ati^ons as ratios of one physiLian for'evei y B^.'^OO 
black citizens as opposed to a 1.7.^0 ratio for whites, or one black at- 
torney for every 6.000 black persons in contrast to \\ 1.6.10 ratio foi 
whites, prompt suppoiters to advance the notion that it is in the 
state's intcre-t Ic ^uppoi f aggressive aflirniative action piogiams. 
Benign or "colof Wind" methods are viewed as totally unrealistic, 
thdre are no acyaphorous leniedies. Given tfie fact that evidence of 
beneficial impacris now appealing, *suppoiteis view the ulea of be- 
nign neglect as tiagic Moieovei, supporters aie vociferous in then 
distinctions between gOcils and ijuotas. The former aie amelioiative 
and the latter lepugnant, goals aie only goals— target^ to aim at 
through good-faith efforts with nioie consiilei«ition given to qualifi- 
cations or possible remediation than tiv a nicie numerical count. 
Since bright critics a^e having difficulty making the distinction 
between goals and ijuofas. it is felt that theie is little sinceiily . only 
the opportunity to u^e the teim as a cuilgel to strike down tlje total 
affirmative action effoit. Moieovei. in teims of cilucatioifTt is felt 
that the univeisitv shoulil be a microcosm of thc^ociety of which it 
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is a part. Therefore, the representation of minorities and women m 
universities is in the interest of the state, the university, and its 
student body. In regard lo discrimination, reverse discrimination i§ 
thought to be benign, temporary, and possessing one important 
characteristic that distinguishes it from traditional di^scrimination: 
It is employed by the majority to include rather^'than exclude 
minonties in the realm of human activities that have been enjoyed 
by the majority, particularly white males (16). Obviously, for af- 
firmative action supporters, a ruling for Bakke would be viewed as 
having disastrous consequences. 

In this author's view, a decision for Bakke would be a serious 
■blow to the progress we have made thus far as well as a reinforce- 
ment of the growing belief that a national backlash is now taking 
place. Unfortunately, the greatest impact of such a decision may be 
■on those 15 or so federal domestic programs designed to give 
assistance to those who need it most. For example, the S4 billion 
public works law passed by Congress last spring in an attempt to 
stimulate empfovment in the construction industry requires that 10 
percent of the money for each construction project be spent on 
purchases from minority businesses. Just a month ago. the Adminis- 
tration ordered government agencies to double purchases from 
minority-owned businesses during the next *.wo fiscal years (27). 
Moreover, several states and cities have been required by the 
federal courts to add more racial mi^norities to their payrolls with hir-- 
ing ••atios being a part of the order. It would seem that all of these ef- 
forts would be in serious jeopardy. 



Conclusion 
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In this paper. I have discussed only a few of the problems facing us 
in our efforts to advance equality of opportunity. I realize also that I 
have not mentioned problems within two areas of my own research 
interests— the effectiveness of collegiate compensatory programs 
(there are many, and they no longer can rest on the beneficent 
characlei of their mission but must be held accountable for their ef- 
forts) and the problems of admissions tests (I am deeply disturbea 
over what appears to be an ever-increasing level of importance given 
to small score differences on graduate school admissions tests where 
the competition is fierce and the decision is often relegated to a mere 
quantitative ••pecking order.' ). However, it would seem that among 
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the many aspects related to the social implications of the pending 
court decision, it will be its broad imp* ct upon society that will be of 
major interest to future historians. Future observers may conclude 
that the height of this nation's greatness may be measured by the 
depth to which we refuse to allow our most disadvantaged citizens 
to descend. 
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Educational Tesiint^ Service 

A good many people are apparently very much concerned these 
days about tests and testing piactices, some pt opie even think there 
ought to be a law . Bills arc being intioduccd m both state and federal 
legislalures to control testing practices. Congressman Marrington, 
for example, is sponsoring a bill that, among other things, v\ould re- 
quire putting a warning label on all score repoits stating that 'These 
• scores are approximations" and including a repoit of the standard 
error of measurement. 

The National Education Association has recently reaffirmed its 
opposition to standardized tests, and the report of the Nr a task 
force pr-ovides som.e reasons for rts st*nui. Tests avc thought to be 
deficient because they measure cognitive k<irnrng to the excli|>K)n of 
emotional and physical development, they ^ .nah/e creative think- 
ing because of their heavy emphasis on multiple-choice rtenis, and 
they are often culturally biased. I hereaic other frequently vorced 
criticisms (hat ni a niighl have included, such as that tesK are 
coachable, that they influlMice the curriculum, and that they predict 
only nar r ow academic cr iteria and not cateei success. 

Some people in the testing business tend to reject such criticisms 
out of hand by saying that the critics don't reall> undei stand the 
problems, or that any f*urlts of testing are attributable to misuse of 
tests rather than to deficiencies in the tests themselves. But I think it 
is a good trme for those of us rnvolved rn educational and 
psychological testing to stand back and take <i fresh look at what we 
are doing There must be reasons other than politrc<!l expediency for 
the widespiead pn^est Maybe we ought toconsidei more seriously 
what improvements in tests and testing procedures should be made 
befor-e the law rmposes reslrictrons that make it moie difficult to 
move ahead in the testing field. 

Some of the criticism can be disposed of quite easily Ol ( ottrsc A 
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cognitive test doesn't measure emotional development. Neither 
does a thermometer measure humidity, but the thermometer is still 

useful. . , 

Other criticisms are not so easy tc dismiss. For example, the laea 
of reporting the error of measurement along wiUi test scores has 
been around fof a long time, and some tests defuse score report 
forms that call attention to the error of measarctfient. But the great 
majority of score reports are not accompanied by any kind of 
graphic, numerical, or verbal indication of the error in measure- 
ment True, the test manual may contain a section on error of 
measurement, but it is easily overlooked or forgotten when score 
reports are examined. Why isn't it more common to put the informa- 
tion on the score report itself.' Do we have to be forced by law to do^ 
something that most test experts would consider sound practice : 

In the short time 1 have this afternoon. I can't go very deeply into 
a discu.ssion of which criticisms are justified and which ones are not. 
But I would like to comment on the following allegations about tests: 
that multiple-choice tests penalize creative thinking: thlit tests don t 
predict success beyond narrow academic criteria; and that tests 
influence the curriculum. 



Do Tests Influence the Curriculum? 

Let s start with the charue that tests influence the curriculum. A 
number of years ago. a superintendent of public instruction in a state 
not far from here stated publicly that he favored banning all educa- 
tional t-sts because they influence the behavior of teachers I 
thought he was wrong in wanting to ban tests. If I had been the 
superintendent and I had in my grasp a tool that I could use to 
influence the behavior of teachers. I would have held on for dear 
life Let me tell you why 1 think he missed the boat. 

During Worid War 11. 1 was a staff member of a project, headed by 
Harold Gulliksen. that was supposed to do esearch on selection and 
tr:fining of naval personnel. Among other things, we conducted 
validitv studies of the tests used in assigning recruits to naval train- 
ing schools. One of our findings was that the best tests for predicting 
grades in gunners mate schools were verbal and reading 
comprehension tests. This didn t make much sense, in view of what 
gunners mates are supposed to do. but assignments to service 
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schools were nevertheless made in accordance with the empirical 
evidence. 

Later, our research group was given the assignment of improving 
^ grading practices in the navy service schools, and I was sent to work ■ 
. at the.gunner's mate school in/Ba^nbridgC, Maryland, We found thitt 
the lecture-demonstration methc^d of teaching was used. The 
students studied the technical manuals, and the examinations 
0£ consisted of multjple-choice items based on the lectures and 
manuals. The items dealt with such topics as muzzle velocity and 
the function of the breech block locking bolt. Since the job for which 
these students were being trained was to maintain, adjust, and repair 
the guns aboiird a warship, it seemed more reasonable to use perfor- 
mance tests. Accordingly, we developed a set of tests that required 
students to perform such tasks as adjusting th^ojj^buffer for 
maximum rate of fire on a caliber .50'Brovvning machine gun, remov- 
ing and replacing the interlock carrier spring from a 20 MM gun, and 
removing and replacing^he extractor plunger on a 5"'38 antiaircraft 
gun. The instructors complained that the tests were too hard. They 
were right. Few of the sti(dcnts could perform *he tasks, even with 
liberal time alFowances. 

Since we had orders,^ the performance tests were nevertheless 
given at the end of each unit of training. Because successive classes 
overlapped, the new students soon got word as to what the new tests 
wer^ike, and they began practicing the assembly and disassembly 
of guns. Perform<tnce on the tests improved with each class. The 
instructors also got the point. Tiicv moved out the classroom chairs 
and the lecture podium and brought in more guns and gun mounts. 
The upshot was that students spent most of their time practicing the 
skills rec^uired in repairing and adjusting guns. The tests soon be- 
came loo easy. The vaudity coefficients changed too: The verbal and 
reading tc^t validities dropped, and the mechanical aptitude and 
mechanical knowledge te^ts became the best tests for predicting 
grades in gunner's mate school (4). 

Note that no attempt was made to change the curriculum or 
teacher behavior. The dramatic changes m achievement tame about 
solely through a change in the tests. The moral is cle.^r. It is possible 
to influence teachmg and learning by ch.mging the tests of achieve- 
ment. It is also clear that those who make the tests have a great 
responsibility to produce tests that influence teachers to teach, and 
students to learn, the knowledge and skills that truly reflect the ob- 
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jectives of the training program. 

While I am on the topic. I should mention the related issue of 
coachability of tests. When important administrative detisions 
depend on scores. stud<ints arc likely to seek to gain an ad- 
vantage by attending coaching schools or buying books on how to 
take tests. If teachers think ihcy are to be evaluated on the basis of 
their students' scores, they are tempted to give sample tests for the 
students to practice on. Such coaching is usu^jilly viewed as unde- 
sirable, and no doubt it is undesirable under some conditions. Bui 
note that what went on jin the gunner s mate school was basically 
coaching for the tests. Coaching is untlesirable only if it results in 
improving test scores without increasing proficiency with regard to 
the real instrut;tional objectives. If tests can be made that truly 
reflect the school objectives, the difference between coaching and 
teaching disappears, and efforts to get high .test scores are the same 
as efforts to attain the desired skills and informa\ion. We should be 
making achievement tests that are coachaDle in this desirable sense. 

To summarize: Yes. tests do influence tliv behavior of teachers 
and students. The,onl> question concerns what the teachers and 
students are influenced to do. We should be providing tests that eu-^ 
courage;^ the teaching and Jearning of the knowledge, skills, wwd 
abilijies that represent all of the educational ohjCLtives. ;ior merely 
, those that are .easy to m 'uc with a paper-and-pencil test. 



Do Tests Penalize Creative Thinking? 

Now let's turn lo the eroticism that multiple-Lhoice tes'ts penalize 
creative thinkmg. Presumably this thaige is based on the fact that in 
taking a multiple-choiLC t^st. the e\aniinee does not have to think'of 
the options foi himself, he merely has to read the options, evaluate 
them, and choose the best one. These aLtivitic'y^fqtrire^ompreheii- 
sion of the problem, a baekgi ounil of re Icytlm information, and a 
certain amount of reasoning and judgment, depending on the nature 
of the particular item. Real-life problems moiv often present 
themselves in an open-ended fbim. A problem aiises. and the ques- 
tion is "What shall I do?" The individual nuist then think of at least 
one solution to the piohlem. licttei . he will think of several, or 
many, po^ ible answeis. I iom heie on. the process ma\ be much 
the same as foi multiple-Lhoke pu>blcnis. Reasoning and judgment 
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are involved in choosing the final solution. The^ncccssity of thinking 
of , the options for oneself a'ppears'to be-vvhat characterizes a free- 
answer problem, .and thinking, of the options presumably requires 
such abilities as originality and ideational fluency- the divergent- 
production abilities in Guilford's structure-of-inteilect theory (2). 

Bill Ward and I have been gathering data to find out what 
psychological processes are involved in problem solving, particu- 
larly creative problem solving (I). We have developed a'set of 
scientific thinking tests intended to provide the dependent variables 
for studies of problem-solving behavior. These tests are meant to 
simulate some of the tasks often performed by a behavioral scientist.. 
Thejr titles are formulating Hypotheses, Evaluating Proposal's, 
Solving Methodological Problems, and Measuring Constructs. * 

ril describe Formulating' Hypotheses. Each item is a brief 
description of a scientific investigation. The results of the investiga- 
tion are shown in the form of a graph or table, and the major finding 
is clearlv^stated. the examinee s task is to suggest hypotheses that 
might explain the finding. He is asked to write not only the 
hypothesis jie thinks is mo'st likely to be correct, but also other 
hypothesp that should be considered in interpreting the data or in 
planning^another iiivestigation. The test is not of'the multiple-choice 
form. The candidate must think of the h> potheses. write them dovvn, 
and then mark the. one he considers most likely to be corr . In 
other words, the examinee has to "think of the options for i self 
before choosing the best one. 

^ A number of scores can be obtained from a test composed of such 
items. We have worked so far \i'nh six scores: ( I) the average quality 
of all the hypotheses written bv a candidate. (2) the average quality 
of the hypotheses the candidate thinks are his best. (3) the average 
quality of the hypotheses that are best according to our scoring 
system, (4) the Tjumber of hypotheses vCritten. (5) the number of 
unusual hypo^t^ses. and (6) the number of hypotheses that are not 
only unusual 'but of high quality. Our investigation of the 
psychometric properties of the tests shows that reliable scores can 
be obtained, that the quality scores are not highly correlated with 
number scares, and that the tests are of suitable difficulty for car,di- 
dates for admission to graduate school or for first-yeai graduate 
students. * ' ' ' 

We have also developeil another form of the Formulating 
Hypotheses test that is scorable by a machine. In this version, a list 
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of hypotheses is presented, and the candidate is s'ven two tasks: 
First, he is asked to mark those hypotheses on the list .that he thinks 
ought to be considered and. second, he is asked to choose the best 
hypothesis on the list. Scores analogous to those used for the free-- 
response form are obtained: 

We are now analyzing our data to find out how the free-response 
and '-machine-scorable formats differ with regard to what they 
measure. One finding is that the correlations between corresponding 
scores from the two forms are low. The highest correlation is .33: 
this is the correlation between forms for scores based on the quahty 
of the ideas the candidate thinks are his best. The correlation 
between forms for number of hypotheses is .19 and for number of 
unusual hypotheses it is .17. These low correlations are not at- 
tributable to low reliability. Clearly, the free-response and machine- 
scorable test§ do not measure the ssjme thing. 

Another kind of comparison has to do with relationships of the 
scores to measures of various cognitive nihilities. The results show 
that' the qiialitY scores are related to the same kinds of ability, 
whether the scores are based or free-response or machine-scorable 
forms: Quality scores from both forms correlate with tests of reason- 
ing abilities, particularly inductive reasoning and logical reasoning, 
and also with tests that measure cognitive flexibility— that is. ability 
to change sets in solving a prot)lem. But the two forms differ strik- 
ingly with respect ^to the correlations of number scores with 
measures of divergent production. Only for the free-response form 
do the number and niimhcr of iiniismd responses correlate sig- 
nificantly with divergent-production mea.sures— tests of expres- 
sional fluency, ideational fluency, and originality. 

These findings are based on some preliminary results just otl the 
computer. A more comprehensive analysis \\\\\ be made and 
reported later, But it appears that the ansvver is clear, at least for^the 
two forms of the Formulating Hypotheses test. If you are interested 
only in the quality of the ideas, the fi ee-responsC and the machine- 
scorable test do not seem to iliffer appreciably with uespect to tne 
.cognitive abiffties reouired. But if you are interested in the c'andi- 
date-s ability 4 think of options foi himself, and to.think of options 
that mbstl^th/r candidates not think differences are found- 
the free-response test taps divergent-production skills that are not 
measured by the machine-scorable form. To the extent that tests 
influence the behavior of teachers and learners, free-response tests 
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would presumably be more likely to enhance the learning thexii- 
vergent-thinking skills that-are involved in problem solving. . 




Do Tests Predict Career Success? ^ 

Now let's fleal with ih^ criticism that tests may predict grades but 
they do not predict career success. Good data qn this topic are hard 
to find, primarily because satisfactory criteria oncareer success ate 
not usually available. Most Studies of career success use such cri- 
teria as supervisors' ratings, salary, or* rate of promotion, all of 
which are as likely to reflect h5w well liked or popular the indi- 
viduals are as how proficient they ar^ in whatever kinds of problem 
solving and^decision making may be required of them. If we want to 
know how well tests* predict career success.* we need better criteria 
of success. And this probably means that we must crej^e them. 

The Formulating Hypotheses test and the other scientific thinking 
tests were originally developed to provide criterion variables rele; 
vant to the work of a behavioral scientist; so perhaps we should.see 
how well conventional selection tests predict scores on these tests. 
We find that the correlations of orl altitude and achievement test3 
with scores on the free-response testX of scientific thinking are not 
high. The median correlations of quality scores with the three GRt* 
tests (V. Q. and Advanced Psychology) are approximately .35. For 
the scores based on number of responses, the median correlations 
with GRU scores are .20 or lower. Assuming that, the tests of 
scientific thinking are reasonable simulations of aspects of the work 
of a behavioral scientist. cleari> the conventional tests predict pnly 
those aspects of scientific thinking that involve reasoning, not those 
that require the divergent-thinking abilities presumably involved 
creative problem solving. 

§ome findings from research on medjcal education are relevant to 
the issue of predicting career ^uccess. There is a good deal of evi- 
dence that conventional aptuude and achievement/tests are 
reasonably fjood predictors of grades in the first two years of 
medical school. whenAtudents aie getting heavy doses of anatomy, 
histology, biochemist, y. and so on. But the same tests are poor pre- 
dictors of success in the clinical years. wKen students are learning to 
deal with patients, make differential diagnoses, and develop plans 
for patient management. True, the evaluations of clinical ability are^ 
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based on ratings and on student performance rather than career 
performance. Nevertheless, the failure of conventional tests to 
predict success in clinical training at leyst suggests that they would 
.also fail to predict clinical performance in medicahpractice. 
. Because of the concern of medical school people that the conven- 
tional selection methods m'ay not predict clinical skills, a research 
study was initiated that involves development of criterion measures. 
The schools were particularly interested in the problem-solving 
skills required in medical diagnosis and interpersonal skills of the 
sort required in interviewing^ patient. The study is being carried out 
cooperatively by a consortium of medical schools, trs, and the Na- 
tional Board of Medical Examinei s. 

The National Board^of Medical Examiners has primary responsi-' 
bility fof developing the criterion measures. The diagi>estic prob- , 
lem-.^olving tests they have dqvej^oped are i5aper-and-p^^cil simula- 
tions of what hiippens v^hen a doctor meets a new patient,. hears the 
patient's complaint, interviews and examines* the patient, orders 
laboratory tests, and so on, until a differential diagnosis is made. 
'Wit examinee has opportunities Jo irtdicate the diagnostic' hy- , 
potheses that come to mind, the information he needs to o^'aluatc 
the hypotheses, the hypqtheses he entertains after more information 
is provided, and so on, until the decision is reachecf. The tesjs arc 
appropriate for fourth-year studejits and ^residents^ Both frea-^ 
response and niachine-scofaBle versions of the te^ts have beep 
develo|?ed. 

The criterion tests in the ai*ea of iftterpersonal skills require that 
the physician interview live, simulated pjttients.'Each "patient" has 
been trained to describe his or her complaint, to reveal certain other 
infor\nation if asked, and to show iippropriate affect. Scoies will be 
developed after coding the many behaviors occurring during t}ie 
tape-recorded Jnterview. The scores will reflect the physician'.s . 
performance in a great variety of ways, such as responding to the • ' 
patient's affect, time spent asking questions, time spent giving in- 
formation, time ^erlt listening, number of interruptions, and so on.. 

Given such a sb^r of criterion measured, it seems reasonable that 
the most valid selection tests would bey^unt^rparts of the criterion 
tests ihat are suit^ible for candidaLe^r^vho have no medicaij|aining. 
Educational Testing Service has developed a set of experimental se- 
.lection tests of this kind. The problem-solving tc^ts involve social, 
education, and ecological problenvs. and they similarly provide op- 
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portunitfes for the examinee to indicate what hypotheses he is 
considering at each of several stages in dealing with a problem, what 
^information he needs, what sources of information he may try to ex- 
ploit, and spron, until a final solution is called for. Again, both free- 
response and machine-scorabic formats will be^pmployed. 

.For the interpersonal skills tests, the counterparts of the simu- 
lated patients will be simulated counselees wjth various kinds of per- 
«^nal problems that a candidate for admission to medical school 
cotild deal with. AnalogousVnethods of scoring will be used. The 
tests will be administered to first-year and fourth-year medical 
stuUents, ^ 



What a Test Is 

- , . r 

As you may have noticed. I have been trying to suggest b> my illus- 
, trations that we have too narrow a conception of what a test is. To 
most people, a test is a booklet with a lot of multiple-choice items 
and a separate answer sheet. Let me propose a much more general 
definition: A test is any standardized procedure for eliciting th^ kind 

, of behavior we want to observe, and n easure. I,mean the behavior 
W?reall\ want to measure, not merely something related to it. This 
definition imposes few constraints on test makers. If we want to 
measure spelling ability, we can dictate worcis to spell. If we want to 
measure ability to repair machine guns, we can'present guns needing 
repair. If we want to measure t>iagnostic ability . we can provide op- 
portunities to deal with patients who have problems. If we want to 
measure interviewing skills, we can present someone to be inter- 
viewed. Such tests^ would presumably influence' teachers and 
learners, and be coachable, in the desirable .sense: they would re- 
quire the candidi^te to think of the optionsr^en this is appropriate; 
and they would have a better chance of predicting Career success 
^ than conventiom'il tests! 

Pm sure you arc^all thinking that that is iijl very >vell, but how can 
you test a million candidates a 'yea»' ^ith tests that may requiie ma- 
xhine guns or simulated patients, tests that must be administered in*^ 

/ dividually and that can't be scored with ai) optical scanning device? 
One answer is that perhaps it is time to recognize that assessment" 
and evaluation may be sufficiemly important, at least in some 



05 



Ther» Ought to Be a Law 

instances, to justify much more than a day of testing time and a fee 
of fifteen dollars. If, for example, we consider the cost of a medical 
education; the additional income over the lifetime of a. practicing 
physician that is attributaj^le to his medical training, to say nothing 
of the importance of the life-or-death'decisions made by a physician, 
the cost of conventional testing is small indeed. 

Another answer is related to what I have said about coachability 
and the influence of tests on teachers and students. I have argued 
that it [s possible to malce ^sts that reflect instructional objectives 
more accurately than do conventional tests and that such tests ^ ^ 

influence the behavior of teachers and students in ways that enhance 
learning. If I am correct, it would seem sensible to use tests fo/ 
teaching, not just -for evaluation. Forms of a test could be 
constructed in such numbers and variety that'tHey could be used 
regularly for homework or classroom drill. Students could cram and 
teachers could Voach as much as they pleased. The cost of the tests ^. 
would be justified by their value for instructional purposes. 
N The scores would be uspful in other ways, provided that on some 
occasions the tests were administered tinder standard conditions 
'and records of performance were preserved. The scores could be 
used in monitoring the progress of individual students, fof-schcol 
^evaluation, for counseling, for admission to more advanced training 
programs, and for assigning grades to students. School grades are / 
now the best predictors of academic success in higher education;^ 
school grades based on tests that more adequately reflect the 
spectrum of educational objectives might also become the best pre- 
dictors of career success. • ' 

I Realize that I have oversimplified the issues and ignored some • 
thorny problems and that niy proposals are vulnerable t^) attack 
from several pract caf and methodological points of view.- But I have 
been tri^ijig to make a point. I think we have drifted ifito stereotyped 
attitudes about test:, and testing that have tended to fte'eze tests into 
formats that seriously limit their potential, and I want to encourage 
* you to think of Resting in4he much more general sense that. I have 
described. 1 think the consequence would be better tests— tests that ^ 
exfert' a good influence on t^achin^. That are less suscept^le to 
misuse and misinterpreifStionJ that are viewed as "fair," and that are J 
less likely'to.make people grumble that ''there ought to be a law." 
^ ' I discovered .recently that Professor A.. Lawrence Lowelk who 
was president of. Harvafd when \ was bofn, had somewhat similar 
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thoughts abq^t examinations. Here is a brief excerpt from an article 
hepublishedin ihc Atlantic Monthly in"l926(3): 

The question of studying for mai:ks rather than for knowledge* and the 
kindred matter of crairiming for examinations, are not uninteresting and 
are often piisunderstood. The popular impression ... is that a student 
whose primary object is a high gr^dCvdevotes himself ... to memoriz- 
ing small, and comparatively unimportdnt. points in a course, and 
thereby makes a better showing than a classmate with ... .a larger rc^\ 
command of the subject, ... As the [examtnation] questions are often 
" made out and marked this result may', and does, occur. But if all exami- 
nations were so conducted as to be an accurate and complete measure 
of<he education the course is intended to give .... then there would 
be no reasOF^ why the student should not wofk for mArks, and godd 
reascyi why he should. Tochidtra tennis player for training himself^ it h 
a view to winning a match, instead of acquiring skfll in the game, would 
be absurd, because the t\Co things are the same ... if marks are not an 
adequate measure of what the course is intended to impart, then the 
examination is defective. If examinations were perfect the results 
would comnftand universal re^ptct. and high grades would be a more 
general object of ambition. 
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Introduction. 

The way to make hard decisions with soft data is not. in general, to 
harden the datjj. although that may help: It is to improve the deci- 
sion-making process, the methodology. And here [ think we carf 
learn something from the law. The last decade or two have seen the 
emergence of new ntodels of legal reasoning, in particular the so- 
called New Rhetoric approach. In thre same period, the discipline of 
evaluation has also taken on a new form orfiwms— it has become in- 
dependent 6f testing and measurement and it has spawned its own 
new models, the Stake leactive model, cipp model, Eisnpr's con- 
noisseurship model, and so on.. Evaluation is a new discipljwie th'at 
haV> emerged from testing in somewhat the way that nevv law 
emerges from socioTi5^ and demography. What I propose^todo in 
these few notes is sketch jn some arguments for supposing first, that 
this emergence Js truly a* revolution and second, thai (here is a 
sinii!c*r logical structure undcrl>ing both legal and evaluative reason- 
ing—at least, up to a ^oint— a structure tha^t goes far beyond the 
logic of quantitative inference that guides the measurement field, 
and t^hat the essence of this structure is quite well expressed in the 
phrase "making hard decisions with soft data." We should, 
however, amplify, the formula slightly, for hard decisions are just as 
hard, even if the data are hard if thee (^////cc tion (the relevance of the 
data to the decisicms) is soft, as ir often is. 
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Making Hard Dtclsions with Soft Data V 
The Way Things Were ' 

The scenario thplia^ unfolded on the methodological stage in these, 
test yeans ha9<ntroduced^a radically different cast,of characters. In 
.1%0, theSr(Kial sciences appeared oji stage like a chonus of ibm 
junipr management types— almost uniformly dressed in neat but 
boring replicas of the costume favored by their elders and betters, 
the Physical Sciences. That fellow down, the end of the line there 
does look a litfte out of place— collar askew, shoes muddy and not 
tied— that's Anthropology, -apturflly 'Ciiltiiml Anthropology. His 
older brother, Physical Anihropology, blends in perfectly, hgwever. 
Nothing basically wrong with the family. If you look closely, too, 
you might wonder about the chap nearest you; he's prifnped up so 
exquisitely as to raiie soir\e doubts about his reality— that's Eco- 
nomics. Bang in the middle of the chorus, we can see a fine^oung 
fellow with. a familiar look— why, it's Educational Tests and 
Measurements! What really impress6^ one about the whole bunch is ' 
their spirit, 'though. Just bubbling over with pride and enthusiasm. 
Reminds one a bit of that Up With People, show, don't you think? 
Another young fellow ovcy there catches one's eye iis an up-atid- 
comer;, what's that label say? Scientific Jurisprudence, by George! 
Definitety gives bnc a feeling that all'.s right with the worid— and go- 
ing to be better— with this splendid teym ready to pick up the 
burden. Of course, there aren't any ^f/r/.v in the show, but you can 
certainly visualize them doing tremendously valuable things off- 
^age — sewing the costumes, cleaning up, and whatnot. ^ 
' Well, it was a jolly good show. Vvhat was it called? Let's see— the 
Newtonian Follies of the Mine^ecn-Fifties. Still running in the Six- 
ties, too, but they never changed the namcr-can't get the same 
alliteration, you know ^ And besides, the show itself never changed, 
so why bother? 

Some of those young chaps were definitely born for greatness; like 
that Economic,s fellow. Went on to become a real star: they set up a 
special Nobel Prize for him, I believe. ^ ^ 

•That was the/how, folks, and now it's folded, though not ^11 the 
characters hav'^ quite realized- it. What really happened was thai 
people couldn't quite believe it any more, and if people don't believe 
in fairies, why, tfiey just die. Its truly sa^ But then, life must go on, 
and we have to grow up and give up our fairy tales. 

The Methodology TheatVe today has a really different show. No 
chorus line. Some w/fmen and minorities and even some oldsters on 
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the stage. Even th^ Anglos aren't dressed like eSch other. There are 
family groupings, but they're nqt easy to spot, and seem to be;^ 
changing all the time. Truly bizarre characters have emerged, like 
that weird entity they call Policy Analysis, and the feisty newcomer 
So<^iobiolOgy. The Director says it's ^•experim.ental theatre," 
whatever that means. He says that playwrights today feel they have 
to start all over, that the old tradition isn't just dated, it's false. 
says that the triumph of Economics now look<i like the triumph of a 
bunco artist, with the Swedish Academy in the role of suckers. The 
magic of matlt models bemused them; the internal precision of the 
inferences kept them ffom noticing^hat the predictions were no 
good, the* explanations were so facile as to be fraudulent, the con- 
ceptual schemes so frail or so fuzzy as to be fakery . ^ 

Now Tests and Measurements (T&M) was never a' leader in that 
cast, but it suriJiy was misled and it did its share of misleading when 
the show was riding high. It's worth examining its situation a little 
more'carefully. 

I don't suppose anyone would argue strongly for Tests and 
^ , Measureijtients as any more than a set Of tools, pf means to ends. 
But, like their ally Statistics, they soon generated their pwh energy, 
thpir professional associations, their entrepreneurs and publica- 
tions ;and.even their lobbyists in one or another (and finally in many) 
centers of power. The comparison :with paleography with its nu- 
meric^^l, taxonomy in which the quantifiers got out of hj\nd is 
illi^minating, as i% the analogy with mathematical economics. The 
substance of science was too quickly identified with the presence of 
quantitative variables and (more or less) operational definitions of 
them in terms of measurement procedures. Well, there's nothing 
new about the suggestion thai tog many crimes were committed in 
the name of norm-referenc(?d testing, or analysis of variance, or 
, market models. Let's try to be more constructive. There areMwo or 
three disciplines that point to an alternative approach, a more 
promising perspective for redemption. I have in mnda(coun(ifii}, 
jurisprudehce. and evaluaiion. 



The Way ThIngs.Should andShouldn't Be ^ ^ 

If you look at what has happened to the General Accounting Office, 
or HEw's Audit Agency, or California's office of the Legislative 
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Analyst, what you'll see is the massive dequantification of some 
parts of the job of some members of the accounting profession. It is ^ 
a far cry from gao*s original mission to the investigation of the 
Tonkin Gulf incident — from auditing the books to the assessment of 
foreign policy decisions. Yet gaO has acquitted itself well at both 
ends of this long spectrum, as has Audit Agency. By and large, th'ey 
have made the transition more gracefully than most of the quantita- 
tive generation in the social sciences, which still clingjj to an inap-^ 
propriate model. Why? Perhaps it's'because they see the practical 
necessity for change — the need of the decision makers for relevant 
research — and they ; re less diverted by the will-o'-the-wisps of a 
Newtonian synthesis. i\ great world model turning'on the bearings of 
mathematics>Or periiaps it's because the quantities with which the 
accountants were preoccupied always had a highly pragmatic ^ 
referent — money — while the social scientijit^ has never quite been 
able to swallow the idea that true costs have any right to be regarded 
as legitimate variables. Least-of^JI the bulk of economists, whose. * 
capacity for cost analysis is on a par with their interest in it. (The ex- ^ 
ceptions like Hank Levin stand out in ^uch solitary splendor as 
instaTitly to validate the general truth.) 

In any case, accountancy has \ranscended bookkeeping and has 
' achieved a modes! capacity for making hard decisions from soft 
data— that is. {r6t\\ all the data other than bookkeepin^j: entries that 
must be sifted in order to achieve a reality -rekued recommendation. 
Nowhere i.s this better illustrated than in Abraham BilofTs brilliant ' 
book Und,ccotintahlc Accounting. 

The situation in the law is very similar to that in accounting. The 
^ effort to quantify the h //(>/( decision m»iking piocess has been more . 
or less abandoned, since i: can be done only, by giving soft data the 
appearance of hard data and by making judgmental decisions into 
mathematical deductions, which instantly generates bad decisions. , 
The subtleties ^of statistics, which can help us express sonic of 
the softness, cannot save us from the eventual nesni for holistic 
evaluative judgment in the sy nthesi/ing step where we have to tnide* 
off such disparate, considerations as the ch»ince of recidivism against 
the possibility of a mistaken verdict. 

This is not to deny the colossal p,nver of simple math models of 
decision making in the Meehir Dawes tradition— mod^rls that make a . 
shambles out of most claims for the ineradicable necessity of human 
judgment throughout the present territory. Most standard clinical 
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and legdl judgments could be made better (especially given that 
'equity is-a value in itselO by a formula. But not all of them and espe- 
cially not al! of ihe really important on«s. For those, we need judg- 
ment, but judgment, not capricious judgment; and we should' 
face the fact that the task ^f the social sciences in education is to 
imprave that training, uncover arid defuse the biases that operate, 
routinize what can be routinized. and assess the effectiveness, 
reliability, and validity of wjjat's left. 



Training, Not Just Testing ^ 

This move towards training and not just testing, touards improving 
the human instruments and not replacing it. 'can be seen in the 
spectrum of activities at lis in recent years, and it can be seen in the 
new pattern of Supreme'Court decisions since the desegregation de- 
cision. For all the effort to\\ards strict constructionism, towards 
conservatizing the court, it still reaches farther into'the social realm 
than eVer before and in doing so. at on* c demonstrates the difficulty 
of the task and the recognition of its necessity. What ue might call 
social jurisprudence has been edging out the ".scientific'" approach. 
And the pressure that has led to this is not political appointment— it 
is the wider pen-eptlon of reality and its ct)nnections that the latter 
hnlf of the ^Oth century has brought us. The iau 4ias absorbed the 
new precision of the quantitative diViplines uell enough, but it has 
not been overrun by theni. In the mainstreai\iing decisions, in the 
businjy; area, in Scrnuuf-PrU \t. in BokU. theie is no lack of tests and 
measuremen d statistics. But there remains a more important — 
though not J jry well conceptualized — model of law and the 
system bf justice as a social change »igent. a discipline that is not di- 
vorced from cunent evi^nts but is cieating them and also studying 
the way in uhich it is creating them, self-refcrent jurisprudence. 
^ This last feature is one of the most important distinctions betuecn 
traditional conceptions of fests and Measurements " the dis- 
cip'ine of evaluation imdei which f&M must now be see »is largely 
snbsumed' This very conference shoushow far testing has followed 
the judicial system in becoming a ^.hange agent and becoming awaie 
of it. But what part of the trdinin^ in educational psychology has 
been shifted to the study of the social impact of testing? As yet. not 
much. Like testing, evaluations often have effects th.it are larger 
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than those of the program being evaluateck not hard, because few 
programs work, but not easy, because few evaluations are effective. 
Now t&M has long been aware of the problem of ^'reactive** or ob- 
trusive measurement. Evaluation is extremely aware of reactive 
reporting as well and has long been studying and improving' The - 
report. \\ w^is also common enough in the social sciences Co qotice ^ 
self-fulfilling or self-refutin| prophecies, a kind of self-reference: but ^ 
it was not common to take seriously the idea that it was an obli^a- 
turn of the''investigati)r to investigate the investigation. The environ- 
. mental impact research of the field biologist mu^Mso be studied by 
that same biologist for its effect; this is the intrusiveness imperative, 
the limiting case of ^ A/zti/M/n^'. self-applicati?)n of the scientific 
method. Evaluation has tfiis very strong and distinctive empha^js on 
Xhc vUiliuition of evaUuitions, This has also leJ to a sterner and 
wider view of reactive measurement, not only with resp'^ct to pro- 
tecting privacy but in simpler ways: The very iJea of a four-page ^ 
m^^t^ument for the stuJentN evaluation of teaching is now seen as • * ' 
absurd because of what it does to response rate and response 
stereotyping. ^ - 

There are many othei respects in which contemporary evaluation 
has grown a long way beyond its quantitative ancestors. It has 
developed new dimensions of analysis, never to be found in the 
T&M trainmg programs. (Cost analysis is one of these.) Again. * . 
while It has been avidly soaking up alternative .ipproaches to causal 
investigations, it has also totally transcended the notion that evalua- 
tion IS either necessarily . typically . or principally a matter of causal 
investigation at all. The evaluation of curriculum materials for sexist 
bias or factual error via content-analysis techniques is no! a causal 
• investigation. The matching for congruence of test items with course 
* objectives, the analy sis of the sequencing of curriculum modules in 
terms of logical progression— these are*noncausal evaluations of 
/ tests or curricula (i^ very large parts of such evaluations in some 
7 evaluation contexts). Fhe creation of correct descriptions of com- 
j plex programs may be tne mam task of an evaluation; it is not causal 
analysis. Looking for injustice in a school's admis.sion or^ disci- 
plinary system is not causal investigation. In even rnore general 
terms, evaluation has tonic to see that traditional control-group 
methodology N a limited speci«d case even in causal research, not 
because there are better approaches iilvolving no comparisons- 
there aren't — hut because thej/zo-m .////u /z/ 'control is usually the 
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wrong one to use and always the wrong one to use alone. Instead, 
we have to devise or luuover the appropriate iomparison groups, 
for decisions are rarely between treatment and no treatment but 
between two alternative treatments, and one or more of these may 
have to be invented by the evaluator :f the evaluation is to be 
maximally useful to the decision maker. Thus, as with the law, 
creativity as well as study comes into fult play. 

A'gain, ^there -is. the evaluation 0/ tests themselves, the activity 
which truly establishes the dominance of evaluation over T&M. 
Test people are becoming increasingly aware of and skilled in.this, 
as you all know. What they are doing is evaluation, something much 
more general and less quantitative than most measurement. Finally, 
one might mention the extent to which evaluation has turned away 
from accepting the framework of intention, of goals* towards look- 
ing at the actual effecis of a program or product. 



The Need to Reach beyond Measurement 

Rarely has the need for better tests been n\y>re obvious than in this 
meeting where we have reluctanth but seriously considered selec- 
tion for admission to professional schools b> nindomization in the 
absence of better methods. It is clear that the evaluation of tests us- 
ing only the standards of the discipline of tests leads *o some of the 
worst abuses of testing— the confusion of correlation with common 
reference, of aptitude with ability, of criterion-referencing with 
competency. \)f objectivity with multiple choice, or reliability with 
consistency, of internal consistency with validity, of norm deficit 
with needs assessment, of test bias with item bias, of discrimination 
with utility, and so on. We do sorely need to reach beyond the 
measurement approach. Because of the inertia of the professional 
systenS it may lake a more explicit analysis of these errors to do this 
than it should take, and 1 hope i is will take the leirtl in this further 
move towards C(2psciousness raising in the test business. 

When we turn to statistics and its interaction with T&M, once 
again we find the iinLomnron few who have broken the grip of the 
rapidly rigidifying discipline— the Moslcllers, Lights, and Tukeys 
who can play the game when they should hut expose it as only a 
game when that matters. 

The grip of statistics nearly became a death grip five or six years 
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ago when evaluators brought up in that discipline so dominated the 
educational R&D area that only statistical significance was quoted in 
the documentation b> these evaluators of the results of fitld tests of 
educational products. There wajj no mention of raw-score dif- 
ferences. No *needs-assessment data to show that the statistical 
significance was worth an educational Hamn to anyone in the "treat- 
ment" ^roup. We turned up tha^l gem when tjs had the contract for 
selecting the best of those materials for dissemination funding. But 
.even ets can slip: if I recall correctly, its evaluation of Sesame 
Street bnly quoted and certainly highlighted" thfe statistical sig- 
nificance of the intergroup difference. It never Tocus'ed on the 
question: Is the actual difference in gains worth the. cost of the- 
program to the"ta\pa>er. parent, teacher, and child, all of whom 
paid a good deal for the treatment? It turned out that the absolute ad- 
vantage of Sesame Street over the standard approache\was tin\ . al- 
though statisticalh significant because of the Huge number involved. 

Well, so much for the ritual hand-bit»ng activity ; evaluators have 
picl^ed up from jurisprudence the adage that it\ not enough to bite 
the hand that feeds \ou. it's essential that it be seen that one biles it. 



Moving Closer to Real Decisions 

We're by no means out of the sR^idow of confusing: quantity with 
quality yet — perhaps we never will be. But the evaluator today is 
less likely than ever before to be deceived by the tiappings of quanti- 
tative measurement into a conclusion about quality. And she or he is 
certainly outgrowing the temptation to run back and hide in the 
shadows v^hile bleating plahtively that "social science is value-free, 
social science is value-freel** Making evaluative judgments is 
exactly v^hat science is all about, measurement provides data, but 
the use of measurements requires {he ex^tluatu'n ol5 measurements 
and of milch more. What has happened here could be put in this 
way. As measurement has come to be a smallei part of a laiger dis- 
"cTpline. as F.ducational Testing Service has become Hdijcational 
Testing. Training, and Kvaluation Service, as statistics has itself 
come^under evaluation as stuidy or si'ppery oi sloppy, so we have 
come rioser to the world where leal .decision^ Have to be made. 
Made, for example, by judges who spent months reviewing the evi- 
dence for special education classes oi the effect of racial separate- 
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ness or the reliability of nonmoney indexes of quality in an educa- 
tioijal program, only to find that there are no useful wahuitions to 
which they can appeal. As a result of this lack, decisions were made 
which will cost us billions and ma^ easily have to be reverted. In the 
law. there are procedures for mai^in^ decisions in the absence of 
relevant evidence, bs in science. Burlhose procedures would never 
have to be invoked if we had done our job of performing the evalua- 
tions that are relevant to major social programs. We never really 
wanted to maj^e'hard decisions with soft data, so we made the data 
look hard when it wasn't or developed arguments why we shouldn't 
have to make such decisions at all. The law has never had those^op- 
tions— at least it has had them very rarely since the more creative 
days of King Solomon. The law has to make decisions, however soft 
the data or the connections between it and the decisions. 

As a result, the law has developed methods of reasoning that are 
substantially 'different from those of the social sciences; in 
particular, the logic of argument from precedent and analogy, the 
logic of prima facie inference, and the burden of proof. It will pay us 
well to study these and some other, more procedural devices from 
the law. It is no accident that one of the most interesting models of 
evaluation today is called the jurisprudential model b\Bob Wolff or. 
in simpler form, the advocate/adversary model by Bob Stake. One 
way of expressing the latent fiiuction of the adversary model is to 
say that it Ic^^itimates an appropriate ranf^e of interpretatiotus^ 
thereby conveying in a very graphic way the softness of the data. It 
is the evaluative equivalent of adding the standard deviation to the 
statistic of the mean— and it is. of course, a standard procedure of 
tlte legal system. ^ 

In the evaluation :^-area of bias control, to take another 
example, one line of tfiftught has led to the so-called goal-free 
procedure in evaluation—one in which the legiil idea of justice as 
blind is extended one step farther than has been traditional in experi- 
mental design, where double blind methodology was picked up from * 
medical research. ^ 

And so on. The model of legal process is a fertile one for evalua- 
tion. And the model of legal reas^^ning is no less useful. A few years 
ago. 1 had occasion to do a rather dettiiled Lomparison between the 
logical methods (the standards of evidence) m social sliculc and the 
law for Journal of Le^al Lihuation. I couLluded that even the 
striking apparent" diffeiences were not leal. but I was impressed by 
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the novel view of scientific reasoning that was forced on one by the 
detailed comparison. I do not find the New Rhetoric analysis persua- 
sive, but I 'do believe that there is a need for, and high payoffs from, 
new' models and new study of the logic of the faw, as there is for 
evaluation. " \ * 



Conclusion ^ . { ^ 

As measurement has matured, as evaluation'has developed, we l#ve 
come t^see how .much closer w^e/zW mea?ur«menl^nd its usefufal?- 
plication.to evaluation^are moving towards the oth^r practical, deci- 
sion-oriented disciplines of management, accounting, and law. This 
may indeed be 'a devel6pment in which all the social sciences can 
share, not just measurement, and so end a too-sterile period in in- 
tellectual history. The applied leading the pure, the evaluative and 
judgmental leading the value-free^ 
Let it be so! 
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